Working in competition

When is it good to create a competitive environment for a project? When would you assign several persons/teams/departments with the same task and select the best solution presented to you?

I think it is sometimes a good idea, but there are a few conditions:

1. All participant entities (persons/teams/departments/whatever) must be mutually independent. If you create a competition between entities that need each other's help to succeed, you'll just harm their chances of getting good results and increase the time it takes to get these sub-optimal results.
2. There must be a different general approach for each entity. For example, one team might provide a web-based solution while the other might provide a smart-client solution. If two entities use the same approach, you stand much better chances to succeed if you put them together.
3. (Optional) The task should have an intrinsic reward. If working on the task is a reward in itself, or if the results are a reward even if you don't win the competition - then everybody wins, no matter what. An excellent example is a competition of "the best-looking room" in the office. The more you invest, the better your room will look and the nicer it will be for you to work there, regardless of your ranking in the competition.

Can you think of anything else?

Father of two

My wife (a super-hero if you ask me) gave birth yesterday to our second daughter - Tamar. Everything went fine - it went lightning fast and both feel great (well - as great as possible).

I'll be taking a blog-holiday for a while, until we get settled and my adrenaline gets back to normal.

Bloggers ARE rockstars

Roy recently posted his opinion about the decision of U2U from Belgium to avoid working with Israeli companies for political reasons (you can read my own opinion here). Being a famous technical blogger, Roy managed to attract a nice amount of responses, which can be roughly categorized as follows:

1. People who agree with him
2. People who don't
3. People who think he should keep his blog technical and shut up about political issues

I'd like to talk about the third kind of assertments, but let's first start with rockstars...

 I think it's great that rockstars use their influence to make the world a better place. Bono is probably the best example - he's been doing a tremendous amount of work to help those in need, with the highlight being his work to make the Group of Eight Summit write off Africa's huge debt. Had he not been the rockstar he is - he would never have been able to accomplish that (and a lot of other stuff). The thing is that in this work he had to get himself very much involved into global politics, and there are many people out there who completely disagree with his opinions.

I agree with Philip Haack when he says that "Joel is the closest thing the software community has to a bonafide rockstar". As such, I think it's completely appropriate for him (or any other famous technical blogger for that matters) to talk about non-technical things, yes - even politics!

When Mike Stall compared his baby to a finite state machine or talked about his dog teaching him about race conditions I didn't see anyone complaining. Neither did anyone complain when Scott Hanselman tiled his kitchen or when he talks about diabetes.

Bloggers are not just technical - they are first people. They have families, friends, hopes, regrets, ups, downs, the whole deal. I personally like reading a little more personal stuff about bloggers that interest me. I wouldn't want all their posts to be about their personal life, but once in a while, getting a glimpse of it, a small reminder that there is a person behind the words, is nice.

As for the question whether politics is in-bounds or out of it - I don't see any reason why it should be excluded.

So Roy, do me a favor, disregard these single-minded people who can't accept you have things to say other than "regular expressions", "agile", "test driven", etc - SAY IT, loud and clear!!!

I'll be there reading it :-)

About Windows Live Writer, Blogger and Images

I've been using Windows Live Writer for the past week or so. It's really great - I'm able to post in a much simpler and faster way than before, allowing me to post more frequently.

There is one problem, though - posting images to Blogger! For some reason, Live Writer doesn't know how to post images to Blogger. This is particularly annoying, since I've always had trouble with that and I was hoping it would become easier with Live Writer. So for my previous post, I had to post it without the images, and add the images manually from the browser. Not only that, but I'm an IE user, and it took me some time to remember that I've always had trouble uploading images from IE. So I had to dig out FireFox from the programs-that-are-candidates-to-be-eliminated-from-my-computer-on-the-next-clean-up and do it from there. Not nice - really really not nice!

Distance Measures

Definition

Say you have a set of vectors, and you need to define how similar/different they are from each other.

There are many different approaches to measure the distances between vectors. All these approaches must obey (at least) the following basic rules in order to be referred to as a metric:

Given a real-valued function d : X × X → R

1. Positivity: Distance(X, Y) ≥ 0
2. Identity: Distance(X, Y) = 0 <=> X == Y
3. Symmetry: Distance(X, Y) == Distance(Y, X)
4. Triangular Inequality: Distance(X, Z) ≤ Distance(X, Y) + Distance(Y, Z)

The most commonly used metrics

Euclidean distance (a.k.a. 2-norm)

Manhattan distance (a.k.a. City-Block, 1-norm)

Cosine correlation coefficient

(where θ is the angle between the vectors)

The thing is that there are many many more distance measures, each with specific qualities. For each problem you may need to use a different metric, usually based on some empyrical tests.

Additional metrics

Following is a list of additional metrics, for more details about each, please refer to the links at the bottom:

• p-norm (a.k.a. Minkowski distance of order p): same as 2-norm, just replace the 2 with p, where p is a real number > 1.
• infinity-norm: like p-norm with p → infinity.
• Pearson correlation coefficient, Uncentered Pearson correlation coefficient, Squared Pearson correlation coefficient: all are very similar to the Cosine correlation coefficient.
• Averaged dot product: the dot product of the two vectors, devided by the number of elements in the vector. Very simple, probably too simple in most cases.
• Rank correlation methods: non-parametric methods that look at the rank of the values instead of the values themselves.
• Canberra Distance: often used to detect abnormalities, since it has a bias for distances around the origin.
• Chi-square: often used in statistics, to determine how well an observation fits the theory.
• Mahalanobis distance: similar to Euqlidien distance, except that it also takes into account the correlations of the data set and is scale-invariant.

Sources and additional references

http://axon.cs.byu.edu/~randy/jair/wilson2.html

http://en.wikipedia.org/wiki/Distance

http://en.wikipedia.org/wiki/Metric_(mathematics)

http://en.wikipedia.org/wiki/Kendall

http://en.wikipedia.org/wiki/Spearman

http://en.wikipedia.org/wiki/Mahalanobis_distance

http://genome.tugraz.at/Theses/Sturn2001.pdf

http://www.ucl.ac.uk/oncology/MicroCore/HTML_resource/distances_popup.htm

http://www.ucl.ac.uk/oncology/MicroCore/HTML_resource/Distances_detailed_popup.htm

http://axon.cs.byu.edu/~randy/jair/wilson2.html

http://149.170.199.144/multivar/dist.htm

http://en.wikipedia.org/wiki/Norm_(mathematics)

http://fconyx.ncifcrf.gov/~lukeb/clusdis.html

http://en.wikipedia.org/wiki/Chi-square_distribution

Laptop (short) review - LG LS70 3JJE 1.8Ghz

I bought my laptop 16 months ago. It's the LG LS70 3JJE to which I increased the memory.

Specification

CPU: Intel® Pentium® M 1.86GHz (750) ,533MHz FSB, 2MB L2

Memory: 1024MB DDR2 533MHz (Dual channel) - 2048MB (increased from the standard 512MB)

Screen: 15 " XGA (1024x768), High Brightness: 200nit

Graphics card: ATI Radeon X600 64MB

Hard drive: Fujitsu 60GB (SATA) 5400 RPS

DVD-R/RW: DVD-R/RW Super Multi support most burning formats

Wireless: Intel® PRO/Wireless 2200BG (802.11b/g)

Modem: 56Kbps

Network card: 10/100/1000 on board

Card reader: MMC, Secure Digital

PCMCIA: Express Card/54, PCMCIA Type II

Audio: High Definition Audio (24 bit)

Connectors: S-Video,RJ-45, RJ-11, VGA, USB 2.0x4, MIC-IN, IrDA, S/PDIF, IEEE1394,PARALLEL

General impression

This laptop is a charm. Its specification is comparable to the IBM T42 and at a significantly lower price it gives it a pretty good fight. It's fast, excellent screen, all necessary add-ons are build-in, smooth install. I'm really happy with this buy. The only thing I would have done otherwise, on hindsight, is buy it with 2GB memory instead of 1GB.

Pros

• Fast - the CPU is one of the fastest in the pre-dual-core era, fast memory access, etc.
• Excellent graphics (card and monitor)
• Smooth installation - the CD that comes with it includes Windows XP Pro and has everything you need in it. I installed it from scratch with it, and it was really easy - no need for any other external disk.
• Every possible connector, including FireWire, 4xUSB 2 ports and SD card reader, all of which have been very useful to me.
• Not too heavy, if you consider the size of the screen (and the alternatives)
• In Israel it's sold with 3 years warranty (1 international). I didn't need it, but it's important to know you have it (and worth another 100-200$)
• If it matters to you (it doesn't to me) - it has a cool look. Many people at the university started asking me questions about it only because it looks neat.

Cons

• Expensive - although you're given excellent value for your money, it's still not a cheap deal, even today. So if you don't need all these specs, you can certainly find something that will suit your needs at much cheaper.
• Heat - It heats up pretty fast, and when it does, it's really hot. It does cool down very fast, once you turn it off, but still. Working with it on your knees, especially on a hot summer day, it totally out of the question.
• Hard drive is too small. A laptop like this is aimed at the enthusiast, who certainly need much more disk space than the poor 60GB.
• Graphics card does not provide 24bit colors (16 or 32, but no 24). My home monitor is a Samsung SyncMaster 910v, which has the best output when given 24bit colors. I know this sounds weird, but that's the way it is - when I checked with my old PC, when the graphics was configured at 24 bits the output was better than with 32 bits. Not the end of the world, just annoying.
• Sound is rather weak and not very rich. Buy some good external speakers and you'll be fine.

All'n'all, it's a really good machine, that even managed to surpass my already high expectations. I'm pretty sure that next time I buy a laptop, LG will be on the top of my potential brands!

Testers are from Venus, Programmers are from Mars

I don't know how this works in other parts of the world, but here in Israel it is very common for wannabe programmers to start with a job as testers. Usually it's some part-time job they do in their last year of studies, with the hope of being appointed a fix job as programmer when they graduate. I didn't go through this path, but many programmers I know have.

My impression from this process, as an outsider who has worked with experienced QA Engineers as well as Software Engineers from various backgrounds is simple - it's WRONG WRONG WRONG!!!

Why?

Well, first, it emphasizes a common misconception that QA Engineering doesn't require specific qualities, something most person with enough brain to eat with fork and knife would be able to do. I think it's degrading one of the most important parts of software development. Being a good QA Engineer is hard, really hard even. It requires specific qualities very few people actually have. When you take inadequate people to do a difficult job - don't be surprised that the results are sub-optimal (to say the least).

Second, I think that there are elementary differences between the character traits required from good QA Engineers and from good Software Engineers. Actually, the qualities required are so contradicting that I don't know any person who would be able to be good at both.

Let me elaborate:

• Patience - programmers are usually rather impatient people. When they feel their computer isn't working well, they'll go poke around the registry and whatever, sometimes making things even worse, just because they don't have the patience to wait for an IT person to help them out. A good tester needs a tremendous amount of patience. They may find themselves doing hundreds of tests, each slightly different from the other, for several days in a row.
• Feedback - programmers like to get immediate feedback, to see results quickly. That's why add-ons like Resharper give you the little green light on the side - so you know up-front your code will compile. Before that, most programmers used to hit the 'compile' button every few minutes, because they want to know - now! Testers can find themselves working on a good release, finding very few bugs. So there is really no feedback for a long time - the fact that they didn't find a bug, is it because the software is indeed so good, or maybe they missed something?
• Ego - programmers are very fond of their ego. We're not like sales-persons, but still, we like being appreciated (hope I didn't offend anyone here...). Good testers are rarely adequately appreciated - if they find many bugs, programmers won't like them (which is a mistake, of course, but that's life). If they don't find bugs, then they inevitably will show up in production, which is even worse. So testers need to be able to swallow their pride often, very often. Something most programmers I know are not capable of.
• Order - finding bugs is not enough. Being a good QA Engineer means you should be able to work in a very orderly fashion, documenting each step in order to be able to reproduce the bug (and give the developer as much information about it as possible). I know very few programmers who are good at this - it's sad, but it's a fact of life we must accept and learn to live with.
• Creativity - both tasks need a healthy amount of creativity. Yet, there is an important difference in the type of creativity required. Programmers need a contructive creativity - finding better ways to build software. Testers, on the other hand, need to find new ways to break the appplication. The difference is subtle, but significant.

I'm sure I'm missing still many other characteristics, but I think the idea has passed. When I think of the really good QA Engineers I know - none of them would make a good Programmer. Same goes the other way - I know of no good Programmer who would be able to be a good QA Engineer.

I strongly believe QA is one of the most important aspects of Software Development. In some sense, it's the most important one - it's the last phase the application goes through before it gets to the clients, the last chance to improve it before you make a fool of yourself. To my great dissapointment, many software companies I know fail to understand this. With companies that are not 100% software (ISPs, networking, applicances, etc.) the situation is even worse - some don't see why they need a QA Team, if they hire good enough programmers...

As I said, QA Engineering is hard. Doing this good is really hard. Finding good QA Engineers is hard. It's important to understand this, and accept it, if you really want your software to be good!

In this spirit I intend to post a couple more posts in the next few days (unless my wife gives birth before I get the chance). I intend to talk about "QA Day" and "Sanity Tests". If you don't know what these are - stay tuned, these are very effective tools for improving the whole development process.

SQL Injection Attacks

Scott Guthrie posts about Guarding Against SQL Injection Attacks. He also points to a great post by Bertrand Le Roy on the exact same subject. Oren Eini tried to tried to create a HQL injection with no results so far.

I think that of all, what fascinated me the most was Rocky Heckman's webcast, where he displays a step-by-step SQL injection attack, which cut the air of my lungs the first time I saw it. Be sure you don't miss it!

Moving to feedburner!

I'm moving my RSS feed to feedburner. Please make sure you register to: http://feeds.feedburner.com/WhyNotilanAssayagsBlog . I think I succeeded in redirecting feeds from the old RSS/Atom urls to feedburner automatically. If you experience any problem - please let me know.

The Multi-Tasking Myth - Revisited

Jeff Atwood strongly recommends to avoid multi-task working. He even backs it up with studies and citations from people who have already proved they know what they're talking about.

In principle, I agree with him, but I would like to set one exception (in French they "it's the exception that proves the rule"). The exception refers to research projects. Very often, I find myself immerged in some aspect of a research. I encounter some problem, but no matter how hard I try, I just can't find a way out. I'm stuck! When this happens, the best way out (at least for me), is to step back. I give my brain a rest, letting it work on other things. This could be anything - some simple programming task, reading a book, whatever. The important thing is to do anything that's as far as possible from my initial task. Then, after a couple of hours, it hits me (usually when I'm in the shower...) - I suddenly understand what I've been missing, and what I must do to get out of my current situation and move on.

So as far as I am concerned - when I have a complicated research project to work on, I always have a backup, low-priority task, preferably a very simple one, to which I can switch in case of stall. The switch costs, of course, but at least I don't find myself investing a lot of energy getting nowhere.

In all other cases, context switching is a huge time-waster and usually causes more harm than good, no doubt about that!

P.S: I'm not talking about literally doing 2 things together (talking on the phone and surfing for example). This is something that always degrades the quality of both actions. I admit - sometimes I do that, but then I'm willing to pay the price...

Wireless Security - the difference between Alfred the Optimist, Luba the Ostrich, Uninformed Max and Sacha Paranoia

Many people use wireless routers at home. It lets you use all your computers without the hassle of having cables all around the house, it's cheap, easy to install, and usually works pretty well. Very few people, however, think seriously about security issues regarding their home wireless networks. Even those who do think about it, tend to think their network is made secure by using some relatively-easy-to-hack tricks. Others, not concerned by security issues, might have half their bandwidth used up by their neighbors without them knowing it. The most shocking part of all this, is that many enterprise IT managers do the same mistakes with their enterprise wireless networks. My purpose today is to shed some light on the vulnerabilities and how they can and are being handled.

First, let's remember that a wireless network is, well - it's a network and it's wireless (duh!). Being wireless means that as long as you are in range, you can read packets sent from the access point and send packets to it. Whether you'll be able to understand the packets or whether the access point will do something with packets sent to it, is another problem. Yet remember - it's all up there in the air!

Second, let's look at the possible vulnerabilities:

1. Bandwidth "theft"
2. Network intrusion
3. Denial Of Service (DOS)

Bandwidth "theft"

Refers to other people (usually neighbors) using your wireless network to access the internet. Since most ISPs charge by bandwidth, if somebody is using your internet connection at the same time as you are - then you get less bandwidth for yourself. I "stole" bandwidth several times during the war when I was staying at other people's houses (at some point I started wandering around the country with my own router to stop doing that). As far as I know, it's not really called "stealing", since the air is public domain, so you can't claim anything about packets floating around. But still, if you have someone constantly using your bandwidth, it can become pretty annoying.

Network intrusion

All browsers, firewalls, etc. use the notion of "trusted" and "untrusted" zones. The idea is that in your trusted zone you have computers you trust will not try to do you any harm, whereas the untrusted zone (everything else) can consist any computer in the world, including some that might be very mean to you, if you just give them a chance. Before the wireless era, things were pretty simple. Anything inside the company/house is trusted, anything else is not. Yet now, when you can connect to the network without being physically even inside the building, it's more complicated. You could, obviously, configure your firewall to treat your wireless network as an untrusted zone. This is good, and is actually the way to do, if you're using a publicly available wireless network service (say at the coffee shop or in the mall). However, if you are at home, and have your printer connected to one computer and want to print from another computer, it's much simpler to have all your home network as a trusted zone (and that's just one example, of course). But this means, that any computer connected to your home wireless network will be trusted, even your neighbor downstairs (unless you protect yourself, see below).

Denial Of Service (DOS)

Refers to a general way of attacking a target, such that some important resource would become unavailable. The idea is basically to perform a huge amount of communication with the target, taking up all it's resources for yourself, thus denying them from other, legitimate users of the resource. There are various means of performing DOS attacks on a wireless network, resulting in disrupting legitimate usage of the network. Actually, wireless networks are very poorly protected against such attacks. Anyway, I'll keep this issue out of the discussion for today.

 

OK, so we're focussing on bandwidth theft and network intrusion. How do we avoid them? Well, let me introduce you to a few friends of mine: Alfred the Optimist, Luba the Ostrich, Uninformed Max and Sacha Paranoia...

Alfred the Optimist

Alfred believes people are fundamentally good. He might not even realize there are any vulnerabilities in using a wireless network - why would anyone want to do something bad to his network? Why would someone use his bandwidth, if that person could pay for it himself? Alfred installed the router on his own, keeping the factory settings without changing anything except what was absolutely necessary to actually connect to the internet. He has been experiencing some slowness in his internet connection, especially at night, when he knows his neighbor likes to download illegal music files. But, heck, you know, that's how it goes with the internet - sometimes it's faster, sometimes is slower. Yeah, yeah, he's got occasional annoying pop-ups and he has to reboot 3 times a day otherwise it takes 7 minutes to load notepad, but who hasn't a few problems once in a while? (Note: Most Alfred's don't use any firewall for the exact same reasons, and don't understand how the salesperson managed to convince them to buy that anti-virus license...)

Luba the Ostrich

Luba is a computer programmer. She's not an expert in networking, nor in security, but she's got her BSc and understands both the problems of bandwidth theft and network intrusion. She knows there are various protection method, but she's smart enough to know everything comes with a price - if you're going to secure everything on your network, you will have to pay in performance! And Luba doesn't like to pay in performance!!! So she pokes in her router's configuration options and sees it's possible to disable broadcasting of the SSID. She searches a bit about that, maybe tries it out, and then understands that once the SSID is not broadcasted, her network cannot be found. Hey, that's cool! If nobody knows my network exists, they won't be able to connect to it - so I'm safe. Luba knows, of course, that if this was enough a solution, then there wouldn't be so many other options in the security tab of her router. But as we said, she's an ostrich, so she keeps living in denial. One day, Luba's conscience starts to bother her - maybe my network is not secure enough? So she goes back to poking around at her router's configuration settings, and then she discovers you can configure an access list - a fixed list of MAC addresses that may connect to the network. Coooool - she quickly runs "ipconfig /all" on all her computers, writes down all her wireless adapters' MAC addresses and fills the list. That's it - she can now sleep peacefully, certain she's protected against both threats.

The truth is that in most cases, Luba will be fine with either of these options (especially with both). Yes, you read correctly, in most of the cases, not all. And here is why:

1. Not broadcasting the SSID doesn't do anything to prevent someone from connecting to the network. All routers have a default SSID name. If Luba didn't change her router's SSID, then there is a good chance that many people have networks with the same SSID as she has (all aother Luba's out there). So if I used to be connected to a network with that SSID, and I find myself in the area of Luba's house, my laptop will automatically connect to her network, although she doesn't broadcast her SSID.
2. Even if she did change her SSID, that it is not broadcasted doesn't mean that it is not visible. Each packet Luba sends from her own PC holds her SSID (unencrypted). So if I use some kind of wireless sniffer (links at the bottom), I can easily discover any wireless network currently in work, including Luba's.
3. Similar to the SSID, in an unsecured wireless network, the MAC addresses are also transfered unencrypted, so once I've catched a valid packet with my sniffer, I can very easily spoof the MAC address and use hers instead of mine in my packets.

We must admit - there is no real chance anyone will do much effort to run a sniffer and then spoof her MAC address just to steal some bandwidth. So as far as bandwidth theft is concerned, with both SSID broadcasting disabled and the use of an access list, Luba is practically immunized. However, is she protected against intrusions? Well, partly - if the intruders just try to enter any wireless network to create heavoc - they won't be willing to do much effort (there are too many Alfred's out there to waste their time on Luba). However, if she's concerned about people trying to get specifically into her network, then they are most likely to be ready to make the effort, and then Luba is in trouble...

Uninformed Max

Max used to be an Ostrich, like Luba. One day, when he had his head deep under the ground, a big fat bull came around. The bull was horny, and, well you know... Anyway, since then Max has become a little bit more cautious (and has some difficulties sitting down...). He uses a specific SSID, changed his default administrator password on his router, has disabled SSID broadcasting and uses a MAC access list. In addition, he wants his network to be secure. Looking at all the possibilites, he chooses the one that seems the simplest, yet secure - WEP. He feels good an cosy with his own secured wireless network.

The problem with Max, is that nobody told him that WEP is sooo not-secure that it's just a waste of time and energy. Look at this and this and please don't miss this...

Sacha Paranoia

Sacha is one of a kind. He changes his SSID once a week and uses a 127 character password for his administrator account that changes each time he logs in. Asking him whether he disabled SSID broadcasting or uses MAC access lists could cause him to rip your head off just due to the insinuation that he might have missed that. Sacha knows everything there is to know about WEP's shortcomings. Until last year he was using WPA. Now he uses WPA2. Although his network is as secure as currently possible, he's had problems sleeping at night, imagining minuscule ET's wandering around his precious network. Last week he almost had a heart-attack, when only one of his 3 firewalls succeeded in blocking an attack. Actually, it wasn't a real attack - for some reason his firewall thinks his fax is a malicious software enemy. So Sacha disconnected his router completely. Actually, he's disconnected his computer from the internet altogether. From the power source as well - to be on the safe side. He is now working in his garden, watering his flowers. His hard-disk in his back-pocket, just in case...

 

Interesting software

Netstumbler - For mapping active wireless networks

Airsnort - Can be used to extract the WEP encryption key

Ethereal - Network protocol analyzer

Working at Google

Joel Spolsky points to Steve Yegge's view on Agile. I've personally never had the chance to work using Agile methodologies - it was inconceivable in the companies I used to work for, and now I'm mostly involved in long-term research projects, where development is minimal.

Anyway, I think that the subject of Steve's post is a little misleading. In addition to depicting why he thinks Agile is mostly bad (comparing it to good and bad cholesterol), it also provides a very interesting insight of how it is to work at Google. Like many of the commenters, I think in most companies it's not a realistic approach. Nevertheless, I'm pretty sure every company could learn a few things from them, and at least adopt some of it.

Anyway, I really enjoyed reading that post - I usually avoid reading long posts, but I'm happy I didn't skip this one :-)

Hypocrites

I'd like to add my 2 cents to Roy's post about U2U's decision not to work with Israelis.

The way I see it, politics and business are two very different think, and mixing them together is wrong. The base for this statement is that:

1. That's not the way to settle things. If anything, I think that good business, which leads to good human interaction, could be much more helpful in promoting political ideas than not doing business at all.
2. It's a collective punishment, much like the same things Mr. Uyttersprot is trying to go against.

China is one of the most oppressive countries against its own people - would you stop making business with all Chinese companies, stop buying anything Made in China, because of that?

The American use of Guantánamo Bay detainment camp has been controversial to say the least - would you stop making business with all American companies because of that?

Saudi Arabia's government states that all citizen must be Muslim. The religious minorities are not allowed to have their churches or temples or pray in public. There are also very harsh laws oppressing women. I suggest you'd stop buying all petroleum-based products...

Nike has admitted abusing its workers in the Asian continent. Not buying their products would make MUCH more sense, since it would directly hurt the abuser in question. Yet, I'd bet you'll find at least one pair of Nike's in Mr. Uyttersprot's closet.

I'm not saying Mr. Uyttersprot is right with his political claims. All I'm saying is that there are many aspects to this war, every Israeli agrees with some of the decisions taken during the war and disagrees with others. Like Roy, I believe it's almost impossible to pass a judgement when you haven't been there, let alone when you don't know all the facts. In any case, and even if you do subscribe to the political views of Mr. Uyttersprot, I don't think it's wise to mix business with politics. And if you do - please, have the decency to remain consistent with your claims: stop doing business with China, USA, Saudi Arabia and almost any other country on the globe (including Belgium, which has a history it would rather forget, oppressing the people in Kongo). And if you're not, then you're just a hypocrite.

[Note: I was born in Belgium, live in Israel and hold both nationalities.]

Search enhancement - take 2

Yesterday I suggested a way to leverage the properties of blogs to enhance searches. This made me think of some additional enhancements possible, if you try to use the type of content at hand.

Let me explain:

Current search engines employ 3 basic information sources to retrieve the most relevant results:

1. The actual text in the web page.
2. The structure of the text (i.e. headlines vs. simple content, various HTML tags, etc).
3. The structure of the web - we all know about PageRank.

What is common to all 3 sources is that they don't seriously differentiate between various types of web pages (enterprise vs. private homepages, blogs, newsgroups, news channels, e-commerce, etc.). This isn't completely accurate, since it is possible to perform searches that only search in specific sources of information (newsgroups, blogs, etc.), but that's not the point. 

What I feel is missing is an intelligent usage of the structure of each type of web page. 

Some examples:

1. Blogs, news channels - why not implement a voting mechanism (similar to PageRank or other) that takes into consideration the number of talkbacks, the number of registered RSS clients, etc.? Even if a webpage has a low PageRank, if it has a large number of commenters or many RSS subscribers, from many different places, it may indicate that the site is much more important than it may seem.
2. Newsgroups - number of threads, number of users, etc. I don't see many links to newsgroups on the web in general. Yet, some are very active. Then why not use additional, newsgroups-specific parameters to measure a newsgroup's relevance?
3. e-commerce - Many price-comparison web sites allow their users to rate products and write comments about them. I'm sure that the more products are being rated and the more raters there are, the more chances there are that the price-comparison site is a good one. Even more so - products/vendors with many/high rating accross price-comparison sites should be promoted.
4. Professional Magazines - Many give users the possibility to rank and write comments about products (like CNET), others give users the possibility to give feedback about the quality of the articles (like MSDN). Why not use that as part of the retrieval process?

What do you think?

Leveraging blogs for search enhancement

Here is an idea - any remarks/suggestions would be welcome.

One of the most important differences between a blog and a simple web site, is that a blog changes all the time. The same person could post 10 different posts on 10 different subjects. Each would be related to that person, yet the only place where they get all connected together is in the blog.

Let's imagine I met someone some time ago, we had a nice little chat, and I gave him my business card. Unfortunately, when he sent his pants to dry-cleaning, he forgot to take out the card (sounds familiar?). He remembers my first name is Ilan, remembers I live in the northern part of Israel, that I work on my thesis and that I like making soups.

Had he remembered my family name is Assayag, things would have been fine - a search for "ilan assayag" on Google brings my blog as one of the first entries. Yet he doesn't remember my family name...

What would be really cool, would be to search for everything you know about the person, and get some kind of aggregated results. So in this case, that person could search for "ilan soup thesis north israel". The search engine would know that search results from blogs should be aggregated in an intelligent manner. A simple solution could be to look at ALL the post from the same blog as if they were ONE single document (although I suppose you could come up with better solutions). If this would be possible, then I'm pretty sure that a search like "ilan soup thesis north israel" would return the correct answer, even though each element of the query can be found on another post within the same blog.

Note: I know that it would be simpler to search for "ilan blog", which, in this case, would be sufficient, although I'm pretty sure that if my name was something else, such a simple query would just not be enough.

Three Ways to Inject Your Code into Another Process

Look at this amazing article!!! (by Robert Kuster)

Matlab R2006a impressions

Until recently I was using Matlab 6.5 (R13) for both my academic and professional research projects. I tried using Matlab 7 (R14) several times (university, friends, etc.) but the UI was so slow I never seriously considered upgrading.

Along came Matlab R2006a, which I am now using a lot. The UI is still much slower than 6.5, but not slow enough to become a showstopper. Here is a short summary of my impressions of R2006a, with emphasis on the new features interesting to my work:

1. Overal performance:
1. Calculations - I really don't know. Supposedly, some of the functions were optimized (I ran into at least one or two) but I really couldn't tell I felt a difference (though it doesn't mean there isn't)
2. UI - Version 6.5 (R13) was very stable, fast and with a relatively low memory footprint. Version 7.0 (R14) was such a nightmare in terms of UI performance that it was mostly unusable. Version R2006a is much better than version 7. It's not as responsive as 6.5 (especially startup), but it's absolutely usable. Also, it has some cool added features (enhanced debugging, tabbed windows, etc) that makes it worthwhile.
2. Support Vector Machines - It's the first version with an SVM library. It's extendible and rather simple to use, but is slower than SVMLight. Also, it seems to me that in terms of accuracy it is also inferior compared to SVMLight (although I'm sure it depends a lot on the problem at hand). I have the feeling that their SVM support is still at its infancy, and I dare assume it will evolve with their next versions. For now, it's just a set of function in the Bioinformatics toolbox... Anyway, if you want a simple way to use SVMLight from Matlab, try this out .
3. Genetic Algorithms - The GA library is a real kick! It has all the features you can find in other libraries available online, and more. Also, the UI is very nice and easy to configure. It also generates graphs out-of-the-box that are much more informative than those I've been using so far. The downside is that the documentation sucks (especially when using bit strings) and that I had to debug and solve 2 bugs in the original Matlab code (!!!) to make it work properly with bit strings (I may post on this some time in the future). This being behind me, I think it's a very good library in all important ways: seems to do a good exploration of the search space (not too susceptible to local minima), fast execution, easy configuration, cool graphs, etc.
4. Distributed Computing - I only had a short time to test it at a remote location. Yet I couldn't make it work. The architecture is very simple, but the various components kept getting stuck and loosing connection for no apparent reason.

P.S: I'm posting this using Windows Live Writer - interesting to see how this turns out...

Google Image Search API - now available in Chinese

I'm happy to say that my little article on CodeProject has been translated to Chinese. I know several languages, but Chinese is not one of them, so I have no way of judging the translation :-)
Anyway, I'd like to thank hidecloud for the efforts and time put in this, and I hope it will now be helpful for some more people.

Windows Updates Has Got Some Nerve !!!

Windows Update is a great, indispensable, tool. With all the threats around the Internet, using a non-updated machine is simply stupid.
Yet, it is lacking one huge problem - configurability !!!
The configuration options are so limited I don't know whether to cry or to laugh.

I am using a server to run various CPU/memory/time-intensive tasks. Last week I was running a task that was taking 100%CPU for about a day and a half (and it had several more days to run), when suddenly "poof", the server rebooted by itself. I looked at the monitor in the morning, unable to understand why I get the logon screen, when I knew I had this quite heavy task running. A quick look at the Event Viewer showed me that the computer rebooted at 3:30 AM. Right before that, I see the Windows Update Agent, prouding himself of having finished downloading updates and being ready to install them. Since this happened to me already a long time ago, I knew exactly what happened - my server is (sorry - it WAS) configured to run the updates automatically, and some of the updates required to reboot the computer. So Mister Windows Update decided, without asking my opinion, to simply reboot and kill everything that's on its way. Mister Windows Update is so vain that He doesn't even need to let you know He did it - if you really must know, just figure it out implicitely from the Event Viewer.
BTW, when I said it happened to me a long time ago - it wasn't really to me. It happened at a customer (very large investment bank), suddenly their server rebooted in the middle of the night, without prior notice. Since the server was dedicated to the application I was responsible of, I had to figure out, what was going on remotely (I was in Tel Aviv, the server in London, and many security constraints in the middle). It took me much longer back then, since I had no clue what was going on, and like a polite developer my first assumption was that something is really wrong with my software. It took several iterations, me feeling guilty and stupid not to know how bad my own software can behave, until I found the real culprit.

Why can't we configure things like this (each, of course, should have many possibilities):

• Reboot only for high-risk security updates
• Announce the reboot X time in advance, both on the machine and by mail/SMS/whatever
• Download automatically only updates related to X,Y,Z applications - the rest doesn't interest me
• Install automatically only security updates - the rest leave to me
• Download at time X, send email/SMS/whatever in order to give the administrator the time to run the install in an orderly fashion, if he doesn't then install at time Y
• Before installing, send a message to the currently logged-in user, to let him overrule (i.e. postpone the install), send an email/SMS/whatever
• Install/reboot only if the computer has been idle for some time
• If you are configured to reboot, and really have to kill the rest of the process population, have the decency to let a note behind, in the form of mail/SMS/Event Viewer/whatever

I can go on forever, but you get the point.

Some interesting search engines

A couple of weeks ago I posted about a way to get more results from your search engines (i.e. beyond the number of maximal results for a query). I then referred to Ask Jeeves, which has been along for quite some time, and provides a nice query refinement interface.
I was asked whether I know of any more search engines, which provide more features than the features available by the 3 giants (Google/Yahoo!/Live). Well, here's my personal list (in no particular order):

A9 - Simultaneously runs your search on various data sources. For example, you can enter a query and it will search for images, books, web and wikipedia all at once. The results are then presented, all on one page, but in a separate columns for each data source.
Vivisimo - Clustering of query results into groups of related results.
iBoogie - Clustering of query results into groups of related results.
Mooter - Clustering of query results into groups of related results. The clusters are presented in a graphical form, and when you drill-down you get a similar tree-like interface like the others.
DogPile - Simultaneously searches all major search engines (Google, Yahoo!, MSN, Ask Jeeves and more). The results are presented in aggregated form, with an indication of which engines returned it.
KartOO - Clustering of query results into groups of related results. The clusters are presented in a graphical form. Their UI may seem somewhat complex, but it's pretty cool once you get used to it.
WiseNut - Groups related results into categories. It generates much less categories than other similar engines.
Infonetware - Clustering of query results into groups of related results.

If you have anything to add, please let me know :-)

What is my mother tongue?

I was born in Antwerp, Belgium. Although the local formal language is Flemish, at home and with my friends I spoke exclusively French. Most lessons at school were in Flemish, and so was the language we had to use on the streets.
At the age of 13 we moved to Israel. With my family I kept speaking French, but everything else was Hebrew. Well, except for my baby sister, who was 3 when we moved to Israel, so with her I speak a mix of French and Hebrew. All my studies from high-school to Masters Degree were in Hebrew.
For 10 years now I have been working with computers, all the scientific and other professional material I read is in English, 99% of my mail communication is in English, I like to read books in English, etc.
I type at least 10x faster in English than any other language.
I read the fastest in English, then French and rather slowly in Hebrew.
There is no language I can (hand) write such that other people than me would be able to decipher what I wrote. And that's when write slowly.
When I write (or type) in French, the mistakes I make suit a 13-year old.
I dream usually in either French or Hebrew, but once in a while I even dream in English.
I speak Hebrew with my wife and French with my child.
People say I have a Hebrew accent in French, a French-Belgian accent in Hebrew and a mixed French/Hebrew accent in English.

Now you tell me - what is my mother tongue?

Finding Great Developers

Joel has once again published a very interesting article, this time about Finding Great Developers.
The most interesting part, to me, was his description of their internship and how they guide it to result in perfect recruitments.
For my part I have been responsible of recruiting several employees in the past. In the company I worked for, this kind of long-term pipeline was not an option - I usually had only a few weeks or a few months to find the person I needed, and I always needed somebody with experience.
In my experience, and in contrast with Joel's, employee referrals have always proved to be the best source. The most important part, though, is to interview the referring employee thoroughly, before even starting the process with the candidate. I have always seen that when you talk to someone who knows the candidate, if you ask the right questions, you can get a very accurate idea of what to expect. The clue (and it's IMHO the base for everything related to recruiting) is to ask open questions, that will force the referrer to tell the things you want to hear.
I agree that there is a conflict of interest, when you give a bonus to your employee for a recruit, but if you know your employee, trust her, and interview her thoroughly about the referred friend, you should be OK. Also, although non-compete agreements are sometimes used here in Israel as well, as far as I know, the legal situation here makes it very difficult to enforce them (they are usually overruled by a law for free employment).

Getting more results from your search engine

Most search engines limit the total number of results they return per search string. With Google it's 1000. With Yahoo!, if it hasn't changed lately, it's 5000.
I've been asked a few days ago whether I know of any workaround that makes it possible to get more results per search string. My first answer was - "sorry, no can do". I did find a way to circumvent the limitation imposed by the Google API that limits the number of queries that can be executed per day (which is accidentally also limited to 1000). This workaround is a side-effect of my Google Image Search API. Yet, this does not provide a means to get more than 1000 results per search string.
After giving it some thought, I could figure out at least one way to increase the number of results per query. It's not a very accurate solution, but it's better than nothing. The idea is to use the various search engines that perform query refinement (some call it clustering of results). A good example is Ask Jeeves. When you perform a search on these engines, they also give you a list of suggestions to narrow or expand your search. That is, if you search for "apple", the narrowing suggestions are things like "Apple the fruit", "Facts about Apples", "Apple Tree", "Macintosh", etc.
When you work with this kind of engines (or with a "simple" search engine and one with narrowing capabilities together), you can start out by running the original search (apple) and retrieve all the results available for that. Then you can iteratively retrieve the results for all narrowing queries as well (up to 1000 for each), and keep drilling down as much as you like. Of course, there will most probably be a substabtial amount of duplicates in the results, which you will have to handle. Also, the more you drill down, the farther you'll get from the original query (i.e. query drift). Another problem is that of ranking - say your original query was "apple", how do you define the ranks between the results for "apple tree" and "Macintosh". So this still raises quite a few questions. Yet, in the end, you can end up with a much larger number of results that are to some extent related to the original query.

You may ask - why would someone need more than 1000 results per search string? Besides, the further you go down the ranking, the less their relevance to the original search string. In most cases - you're right. Yet, for some research purposes, not only would you need more than 1000 results - you might even prefer getting these than the "good" results returned in the first few pages.

Can anyone come up with some other (better?) idea to work around this limitation??? If you have an idea - please drop me a line!

(Note: I'm using the term "search string" to indicate a complete search, regardless of the number of results pages you get. The term "query" refers to what retrieves one single results page, since the query also includes the result index at which the results page should start. In other words, all "search results" for a single "search string" are achieved by sending multiple "queries" - if you have 100 results in each results page you need to execute approximately 10 "queries" to retrieve all the results Google provides for that "search string")

Setting default share permissions

If you use shared folders often, you probably know that Windows XP defaults share permissions to "Everyone" (with full access). If you don't know that - shame on you!
I don't know exactly where this default is stored, but if you want to change it, you can do with with the Tweak UI PowerToy.
In the tree on the left, select "Access Control". Then choose the "Default share permissions" in the combobox and click "Change".
Enjoy :-)

Working with Source Safe over the web

I am working (from Israel) for a company from the US (I haven't finished my thesis yet, but my grant is dry and I still need to feed my family...). Since I'm working on applications related to trading, there is a great emphasis on security so every file transfer is done over a VPN.
Lately I needed to work directly with their VSS database. I agree with most that Source Safe is really something that should be left in the past, but it will take some time before I can convince them to switch and in the mean time work must continue. Everybody who has tried to use VSS over an Internet connection knows it's just impossible to work this way. Add to that the cross-Atlantic delay and a VPN link and you get to wait 10-20 minutes only to open a tree in the viewer (that is, if you're lucky enough not to get link errors, which I'm currently investigating with my ISP). So I've been looking for applications that help accessing an existing VSS database over the Internet.
The major tools I found were as follows:

SourceAnyWhere (SAW) - http://www.dynamsoft.com/products/SAW_Overview.aspx

SourceOffSite (SOS) - http://www.sourcegear.com/sos/index.html

SourceXT - http://www.acorden.com/sourcext/features.htm

VSS Remoting - http://www.sourceremoting.com/index.htm

VSS Connect - http://www.vssconnect.com/VssConnect.htm

The prices vary around 150$-250$ per user, except for VssConnect which costs only 30$.

They all work with some kind of web server and have a way to encrypt the data, but in our case it's irrelevant since we are working over a VPN link anyway.

SAW and VSSRemoting have a special feature that supposedly improves file transfer significantly, by transfering only the parts of the files that are different.

We didn't want to waste too much time reviewing each, so we decided to start by checking out the one that looks the most mature (SAW).

In general, I am very happy using it. It works really fast, integrates with Visual Studio 2005 and does the work well. I did encounter a few issues, though:

1. Switching with the same solution between SAW and VSS didn't work well for me. I contacted DynamSoft's support and they tried to help me, but it didn't work. In my case I don't care much, because 99% of the time I'll be using SAW anyway. If you plan on switching between them often - I suggest you check it out thoroughly.

2. File comparison application dissapointing:

a. Compares lines and not words/characters (like VSS)

b. No option to ommit blanks, so even if the difference is just an irrelevant space, you will see the whole line marked as different (like VSS)

c. The GUI doesn't work properly - when there is only one difference, the arrows to jump to the difference are disabled when there is only one.

(NOTE: According to Support, this issue should be fixed soon - in the next release).

3. Sometimes, when I just do something on the solution with no need to interact with VSS, it starts performing all kinds of synchronization operations with the server. During that time I can't do anything with the solution. Since SAW works really fast with the server, it's not the end of the world, and usually this process is over in 10-20 seconds. It also doesn't happen a lot, but still - it's annoying and shouldn't happen at all.

(NOTE: According to Support this is initiated by the IDE and not by the SAW integration client directly)

4. When trying to perform something with too many files (~15+) I get socket errors. The link to the server isn't broken, but the specific operation is aborted. This is a very painful issue, since it requires a lot of manual workarounds. I'm still investigating this issue with both DynamSoft and my ISP - I have reasons to believe it may be caused by problems in my Internet connection. I would still have hoped SAW would be able to cope with minor connection problems, though...

Despite these, we are probably going to purchase it - especially if I manage to fix the socket errors problem by fixing the Internet connection.

As a last note, Support told me that in the next month they are about to release their next version (5). I asked whether, if we purchase SAW before the release, we would be entitled to an automatic upgrade when the next version will actually be released (like JetBrains did when we purchased Resharper 1.5, little before the release of 2.0). I was dissapointed by their answer: they haven't decided yet about the upgrade policy. Therefore, even if I manage to fix the socket errors problem, we will still have to wait for the release of the next version before purchasing (or for them to tell me we will get the next version for free anyway).

The War

On July 16th I had to flee my house – and came back after having wandered around the country with my family for 30 days. If you want to get a glimpse of what the war has done to my personal life, here you go.
I live in a small village in the Izrael valley. It’s in the northern part of Israel, but quite far from the border with Lebanon (about the same distance as Haifa). I have a magnificent little girl, a beautiful wife 7 months pregnant and a crazy dog. In the proximity of our village is an important air-force base. In normal days, the sound of the planes is not very pleasant, but you learn to live with it. We knew, before moving to that village, that if a war was to happen, our little heaven would be troubled, especially due to the proximity to the base, who is an obvious target. We never figured how much …
To keep things simple, I’ve decided to summarize the impact of this war on our day to day life as follows:

• We don’t have any shelter in our house – so from the beginning we had to flee in order to be safe. So we slept 30 days at other people’s who have been kind enough to open their house for us (6 different places).


• Rockets landed a couple of hundred meters from where I was, and more importantly from my girl’s kindergarten (in the town next-by).


• The first time the rockets landed close by, all phone lines in the neighborhood of the kindergarten crashed. From the road we took to the kindergarten it took a while to understand that the smoke came from behind the kindergarten and not from there exactly…


• At some point my wife decided to go back to work (we were staying at our in-laws at the time who unlike us have a shelter). The sirens caught her when she was in the parking lot, ready to go back home. It’s an open lot, with no cover at all. At first she simply dropped to the ground, in order to try and avoid the deadly bullets. When the rockets started to fall she tried to find something to she could shelter under. She found a place with a 3 millimeter roof where a few other people were taking shelter. The funny thing was that the roof was the last of their concern – there were many gas tanks pilled up right next to them … When the attack was over, she literally flew home. Two hours later there was another attack in the same area. 3 of the rockets landed right in the path my wife uses back home.


• One specific attack was particularly scary – we were at my in-laws and when the siren started my mother in law was in the shower. She didn’t make it to the shelter on time. Suddenly the rockets started falling – REALLY close. My wife, who’s 7 months pregnant sat next to me, in the shelter. At that moment, while we felt the whole house tremble and her mother wasn’t answering our calls – I thought we were about to loose our baby.


• Before that same attack, my 2-year old girl was looking at a DVD of the Teletubies. For those who don’t know what this is – they are little creatures who are happy with everything and laugh for anything. You could rip their heads off and they would still find a reason to laugh and be happy and nice. Anyway – from that moment on, my little girl doesn’t stop telling me that “the Teletubies scared her”. She has become moody, winy, and can’t stay more than a few minutes without seeing us both (don’t even think of leaving her with somebody else). She often wakes up screaming shortly after having fallen asleep, probably due to nightmares.


• Until now, when I hear an ambulance my heart misses several beats – my first reaction is that it’s a siren again. Any strong noise (even a door being slammed) makes me fear a rocket has fallen. When I’m with others, we usually exchange looks and it’s clear to all that we all experience the same thing.


• The thing is – we are among the lucky ones. None of our friends and relatives got killed or seriously injured. Just to show you how lucky we are indeed - a friend of us lives in another village next to us. Two of her nephews who live in the same village got hit by those horrible bullets being propelled by the rocket when it explodes. They each got 2 bullets in the arm. In the past two weeks they endured together more than 10 surgeries and they are not over yet. Imagine if the bullets had hit the abdominal region or the head…

SQL Server 2005 - Frustrated by a good feature

I guess that most people who have been using SQL 2005 for some time already know about this. I, for my part, have worked a lot with SQL 2000 in the past, but never had the chance to really work with SQL 2005 until recently. Now I needed to connect to a server on a remote machine and had this frustrating experience...

1. Scenario: trying to connect to a SQL 2005 server from a remote machine fails. It gives the following message: "Sqlcmd: Error: Microsoft SQL Native Client: An error has occurred while establishing a connection to the server. When connecting to SQL Server 2005, this failure may be caused by the fact that under the default settings SQL Server does not allow remote connections."
2. After getting this message once or twice you take the time to actually *read* it, and start checking the settings for the SQL Server.
3. Like anyone with experience with SQL 2000 would do, I opened the SQL Server Management Studio. There I looked at the properties for the server instance and saw that the configuration seems fine and the server is configured to accept remote connections.
4. Back to square one. From this point I started looking for the culprit...
a. Maybe it's the FireWall? I configured the FW to trust my local network, but maybe it fucked up somehow? Disabling the FW quickly showed me that's not the problem.
b. Maybe there is some problem with the SQL version and I should upgrade? Seemed unlikely, yet I checked it out. Turned out to be irrelevant since I was already using the latest version (SP1).
c. Maybe I did something wrong with the connection string I used? Tried all possible variations - nothing worked...
5. In the back of my mind I started to rethink about this error message I got. It says that by default, SQL 2005 is configured not to allow remote connections. I don't remember having changed that - so how come it's configured to accept remote connections? Could there possibly be some other configuration parameter that has some impact on remote connections?
6. I went over all the configurable parameters for the server instance and for the database (from SQL Server Management Studio) - nada
7. Well then, I'm on the verge of throwing my computer out of the window. I'll give Google a last try. Then I found this: http://support.microsoft.com/?kbid=914277&SD=tech
8. It turns out that there is a much more elaborated way of configuring remote connections in SQL 2005. This is done through the "SQL Server 2005 Surface Area Configuration". It is, of course, a good thing they have added this wealth of configuration options - but why couldn't it be accessible from the Management Studio? Couldn't they add an "Advanced" button on the remote server connections options that opens this Surface Area Configuration??? And if they didn't want to put too much things in the Management Studio - why did they give an option to configure remote connections there, when it can't work on its own anyway???

If you ever run into something similar - please remember this, it will save you some valuable time ...

Safely firing events in C# without locking

Long time no see…
I’ve been busy with plenty of stuff, and I don’t think this will change much in the next coming months – I’m trying to finish my thesis. So basically I put my blogging into (very) deep freezing state. I don’t promise it won’t remain that way until I finish with my thesis…
Anyway, yesterday I went to hear Juval Lowy at the C++/VB Users Group meeting. As usual, his talk was excellent and very informative. Also as usual, I learned as much from things he said on the side as I learned from the actual presentation. One of these side-things I learned really stunned me. It’s the simplicity in itself, but I never saw anything written on this, and even when I tried searching a bit about this on the Internet, I couldn’t find any concrete discussion.
Problem definition: How to properly handle events in a multi-threaded environment?
Just to get in sync, look at the following simple event-handling code:
public delegate void DummyEvent();
public class EventDemo
{
…
private event DummyEvent dummyEventRace;

public virtual event DummyEvent DummyEventRace
{
add
{
if (dummyEventRace != null)
dummyEventRace += value;
}
remove
{
if (dummyEventRace != null)
dummyEventRace -= value;
}
}

protected virtual void onDummyEventRace()
{
if (dummyEventRace != null)
dummyEventRace();
}
}

In a single-threaded world, this works fine. I make sure every access to the event first checks that it’s not null, and thus avoid the very annoying exception, and life is pretty cool.
However, in a multi-threaded environment, you could easily get into a race condition:
Thread A wants to fire an event and calls onDummyEventRace() . I manages to check whether dummyEventRace != null and then comes a context-switch.
Thread B gets the context and tries to remove an event handler – it fully removes it from dummyEventRace and since it’s the last handler, dummyEventRace becomes null .
Thread A gets the context again and – Ooops – exception!!!

So, to avoid this kind of problems, what most people would do (me inclusive) was to add a synchronization mechanism, that ensures each of the operations with dummyEventRace would be done atomically. It would look something like this:

public delegate void DummyEvent();
public class EventDemo
{
…
private object eventLocker = new object();
private event DummyEvent dummyEventLock;

public virtual event DummyEvent DummyEventLock
{
add
{
lock (eventLocker)
{
if (dummyEventLock != null)
dummyEventLock += value;
}
}
remove
{
lock (eventLocker)
{
if (dummyEventLock != null)
dummyEventLock -= value;
}
}
}

public virtual void onDummyEventLock()
{
if (dummyEventLock != null)
dummyEventLock();
}
}
This solution solves the race condition issue completely, but introduces a potential dead-lock issue. If the handler of the event must get hold of another resource inside its event handling method, and the same resource is being locked by the thread that tries to remove the event… In other words – it works, but could get nasty.
Anyway, until yesterday, not only did I think that was the proper way to do it – I thought it was the only way to do it.
I was WRONG!!!

Take a look at this:
public delegate void DummyEvent();
public class EventDemo
{
…
private event DummyEvent dummyEventNoLock = delegate {};
public virtual event DummyEvent DummyEventNoLock
{
add
{
dummyEventNoLock += value;
}
remove
{
dummyEventLock -= value;
}
}

public virtual void onDummyEventNoLock()
{
dummyEventNoLock();
}
}
Amazing, isn’t it? The dummyEventLock never turns back to null so there is no reason to check. Thus, no need to lock and you’re rid of both the race condition problem and the dead lock risk!!!
How come this is nowhere to be found in the MSDN documentation (to my knowledge) ???

Matlab, Memory Leaks, Dual-Core CPU and what's in between

[UPDATE: If you're searching for ways to better use your multiple cores, the R2007a release seems to support multithreaded computations. More on this on my recent post.]

I haven’t posted much recently. Well, I can say for my defense that I was sick (actually we were all three sick, each in turn) and shortly after that we went on vacation. Also, next week I have reserve duty, so I’m not going to post much in the near future either. I have still a pending SQL post I talked about a long time ago that is waiting for me to finish the last quarter of it. Shame on me…

Anyway, I wanted to share some performance problems I had to solve recently. They are all related to Matlab, so if you’re not into Matlab I guess it will be of little interest to you J

I recently purchased a very powerful machine with a dual-core CPU, 2GB DDR2 memory, etc. I’m using it to run some very extensive calculations, mostly in Matlab. However, I got very frustrated because:

1. After a few hours of intensive execution, Matlab throws me an “out of memory” exception. And, sure enough, it’s taking up more than 2GB of memory (note that I have 2GB ram + as much cache as I like).

2. I just couldn’t make the PC use all of the CPU. I didn’t expect it to use both CPU’s for a single Matlab process. I expected around 100% usage on one CPU (core actually) and close to 0% on the second. Instead, sometimes I got around 50% on both and most of the time it was around 20-30% for both CPU’s. So I was barely using ¼ of my processing capability (and only ½ of what I expected).

To tackle the first problem I tried everything, read a whole bunch of web pages but in the end I didn’t get anywhere. My first intuition was that for some reason Matlab is not releasing memory it has allocated, probably due to some error of mine, and I just couldn’t find the source of the problem. After some time I started to think that maybe the problem is with the SVM library I am using repeatedly in my process. As I mentioned in a previous post, I have started using a Matlab wrapper over SVMLight. After digging into both libraries, adding some code to track memory allocations, I managed to prove that both libraries don’t free all the memory that they allocate. While working on all this, I learned a bit about MEX-Files (C functions that can be called from Matlab), since the wrapper library is a MEX, of course. So I learned that there are various ways to allocate/free memory when you’re working with MEX-Files. One, of course, is the C malloc/free functions. Every memory block allocated with “malloc” must be freed (other wise you’re leaking), and if you’re using Windows, you must free it exactly once (more causes exceptions). Another problem with malloc/free in MEF-Files is that memory allocated by “malloc” should not be returned to Matlab. An alternative to malloc/free is using Matlab’s mxMalloc/mxFree pair. Memory allocated by mxMalloc can be returned to Matlab. Additionally, when the MEX-File is released (i.e. the function call is over), any memory allocated with mxMalloc that was not part of the returned variables is automatically released.

So, all I needed to do was make sure that BOTH library would perform ALL their memory allocations with mxMalloc, and then I wouldn’t need to take care of the freeing – Matlab would do it for me (as a side-note I must tell you that in my scripts, I call these libraries over and over again in a loop, so there are many short calls to the MEX-File). So I simply aliased malloc to mxMalloc and free to mxFree (remember the good old “#define malloc mxMalloc” ?) and voila – problem solved (“worked around” would be a better definition). I left my process running for 8 hours, and the memory used by Matlab increased only by a few small MBs! Yipee!!!

Once I had my memory problem resolved, I could start thinking about the CPU. So I changed my script files to separate the workload into 2 processes, instantiated 2 instances of Matlab – each running part of the job. At that point, the most incredible thing happened – once I started running the second process, the CPU turned flat (< 5% on both cores) and both processes got stuck with almost no progress at all!!! This drove me totally crazy! I tried changing the process affinity (i.e. define each process to run on a separate CPU), changed some system performance settings and even installed a hotfix that deals with performance problems on dual-core machine (you can download it from links in this thread) – but nada! Then I thought of trying something totally different. The code I was running was located on a remote computer. Actually, it was on a Disk-on-Key that was connected to my laptop (and the new PC has a network map to the DoK). I’m using this configuration because I take the DoK often with me and need to work on it from my laptop. So I connected the DoK directly to my workstation (the new PC with the dual-core CPU) and voila – both processes work marvelously, taking up 100% of both CPUs (cores)!!!

What have I learned from all this:

1. As always, the source of the problems is usually where it makes most sense. With the memory, the problem is with 3^rd party (non-commercial) code. This code was written for research purposes and I guess that as such there was little effort done to ensure no memory leaks. With the CPU – the problem with the issue of working on network mapped disks. I’m sure that once you’ve read everything above you’ve told yourself – “hey, but it’s so obvious!”. So as always, I should have followed my instincts in the first place…
2. My little aliasing trick with malloc-mxMalloc is something to keep in mind. It might not always be a good idea (especially for long-running code) because if there is a leak it remains until the MEX-File is unloaded, but I guess that in many cases it’s a really good and simple workaround with minimum risk to harm code you’re not familiar with.
3. Working from a network drive involves some hidden constraints that must be further investigated. It may be something that is specific to Matlab. Don’t know yet.