Getting information in Matlab

One of the things I really hate with Matlab is that when you don’t know exactly what you’re looking for, you stand a good chance not to find it. Hmm, this doesn’t sound right, let me rephrase: when you don’t know how what you’re looking for is called in Matlab, you stand a good chance not to find it.
Today, for example, I wanted to retrieve the file that implements a specific function. Since I’m using several free toolboxes I downloaded from the internet, I wanted to know whether the function is part of the Matlab installation or of one of the toolboxes. I started by typing:

>> help pca

Which gave me a complete description for the 'pca' function. Yet, it didn’t provide its source. I couldn’t find any allusion to this function in the Help, which indicates there is a good chance that this method comes from an external toolbox. Note however that it’s not always the case – some functions that are part of the installation are not easy to locate in the Help window, even if you know their exact name.
After several minutes of frustration I discovered that the following line:

>> which pca

returns the full path of the file that implements the function ‘pca’ (and indeed it was from an external toolbox).

My frustration was – how the h… am I supposed to know that the function that provides this information is called ‘which’ ??? My situation was even better than most cases, because I vaguely remembered that there is such a function. Sometimes you don’t even know exactly how you can do what you actually want to do, and then looking it out in the Help becomes a real nightmare.

That’s all, just had to get this off my chest…

SQL for Developers Tips and Tricks: 1. Introduction

Lately I’ve been mostly working on research projects related to Machine Learning (both for my thesis and as a private contractor). Therefore, my ability to blog on development issues I’m encountering is somewhat limited. Therefore I have decided to start blogging about stuff I’ve learned and used along the years, both to keep it as a written memory and maybe to help some readers…

So for starter, I’ve decided to summarize various SQL issues I liked/learned/used. Some I got from fellow developers or DBA’s and some I had to learn on my own (sometimes at great personal risk to become completely crazy). I’ll try to do it in an organized fashion, each blog focusing on some other subject. The vantage point is that of a software developer not a DBA. I have seen many cases where a developer coming from C/C++ started using SQL. The first few days she invests in getting a good grip at the basic syntax (SELECT, INSERT, UPDATE, …). Then she starts writing stored procedures and learns some SQL programming concepts (like CURSORs). At some point, she starts getting acquainted with Views and Functions and learns various helpful built-in SQL functions. Then, slowly but surely, the problems start to arise, usually performance. Once the performance issues are more or less under control, there comes a time to design a new database. This new database inevitably includes many flaws, which will have to be dealt with at some point or another.

I have gone through this, and many good programmers I know have gone through a similar sequence. The purpose of this set of posts is to try and give the programmer a set of tools and knowledge base that can help her go through this ordeal with more ease and confidence. This is by no means a complete tutorial, but I still hope it will be of help. If there are any specific requests, please send a comment with the details. If I know the answer, I’ll try to asses it in the future posts.

One last thing before I present the list of issues I intend to cover – since I’m still too short in hands-on experience with SQL2005, most of the information will be provided with SQL2000 in mind. I guess that most of the issues are applicable to SQL2005 as well, although the new SQL version certainly brings a whole series of new issues (with the CLR and all) as well as additional and revised tools. Also, these posts will assume familiarity with the basic SQL syntax (SELECT, INSERT, UPDATE).

Following is the list of issues I plan to asses. This is a preliminary list and may be altered along the way, but should give you some idea of what to expect.

1. Introduction
2. Basic tools you should keep at the ready
1. Helpful web sites
2. SQL Query Analyzer – things too many people learn too late
3. SQL Profiler
3. SQL Programming for Performance – basic programming tips that can help reduce future performance problems
1. Similar code that runs faster
2. Cursors
3. Temporary tables
4. Use of indexes
4. Locks and Deadlocks – using and troubleshooting
1. Locking in SQL 2000 and isolation levels
2. Simple deadlock scenario
3. Troubleshooting locks and deadlocks
5. Troubleshooting performance problems
1. SQL Profiler to the rescue – how to configure, what to look at
2. SQL Query Analyzer - reading the Execution Plan
3. SQL Server configuration
4. Analyzing index behavior
5. Partitioned tables
6. Some advanced Tips and Tricks
1. TBD

Things I do not plan to discuss at this point:

• Accessing SQL objects programmatically (SQL DMO)
• Using ADO/ADO.NET
• OLAP
• XML Integration
• Reporting and Notification
• Scheduled tasks
• Roles and Permissions

Volunteering in Israel

A few months ago I was asked whether I would be willing to volunteer giving free Math lessons to school students. I accepted willingly and am enjoying it a lot. Since then, my wife has started volunteering giving English lessons as well.
I believe it is important both for society and for the person himself to do some kind of volunteering job once in a while.
In that spirit, if you live in Israel and are looking for a place where your skills might be needed, I suggest you check out this volunteering search engine.

C# Code Snippet Library for Visual Studio 2005

I got this tip from reading Scott Dockendorf’s blog (right here).
It’s been some time that the VB people kept laughing at us C#-ers that they have much more code snippets for VB within VS 2005 than we have for C#. Well, not anymore - check this out.
I guess it’s going to take me some time to get to know all the shortcuts, but I’m sure going to enjoy it a lot

Quaero - where are you???

Almost a year ago (April 2005), French President Jacques Chirac announced his backing of a new Euro-centric search engine called Quaero (which means ‘I am looking for' in Latin). The idea is to build a new search engine which should be capable to compete with MSN, Yahoo! and even Google. Tons of very important European companies are involved in this project, as well as the French and German governments.

Mid January 2006, there was some loud buzz in the air about this project. So loud, in fact, that they decided to get into hiding: the small and unconvincing beta website they had in place has been removed and they came with a statement that around the end of January 2006 they will come out with a new statement (actually a release was expected). Look here for a summary of the latest news.

Today is the 15^th of February and I’m still waiting for such a statement…

As a side note, I’m a little ambivalent about such projects. I don’t like the idea of geologically differentiating core web parts such as search engines. I mean, come on! The past decade has seen such an improvement in accessing data by means of the Internet. Why go back? Also, I don’t like the idea of having governments messing with my searches. We’ve seen what they are capable of (see the China-Google story lately). On the other hand, I’m not sure that having companies being responsible for such an important asset being a much better solution. After all, if my interests as customers will not comply with their interests as profit-thirsty company, we all know what they will opt for (see the China-Google story again…).

IMHO there is an urgent need for international legislation of the search engines issue. I have no idea how this can be done (I don’t know much in international law), but that should be the direction. There should be a set of international laws to which each globally-available search engine should comply. Then, an international committee (with concrete punishment capabilities), would be responsible to make sure all companies/governments managing search engines would abide to these laws.

Build a Data Access Layer in less than 15 minutes

To be honest, I still haven’t decided whether I’m an ORM-guy or a traditional DAL kind of person. Until now I have always worked with regular DAL, and due to various organizational constraints, haven’t had the opportunity to thoroughly check out using ORMs in real-life projects. My heart tends to go towards ORMs due to the OO appeal, but my experience with SQL performance issues forces me to remain skeptical until I’ve seen it work.

Anyways, it seems that if you’re still on the traditional DAL side of the map, CodeSmith has a very interesting solution for building strongly-typed DALs in a very fast and easy manner. I’ll surely try this one out next time I have a new DAL to work on!

An extension to Matlab's Image & Video Toolbox

Piotr Dollar (I swear that’s his name) has written a very nice set of extensions for Matlab’s Image & Video Toolbox.
It includes a whole bunch of existing algorithms with serious performance optimizations, as well as some that are missing in Matlab’s toolbox.

Thanks man, it’s really helpful for my MSc research project!

AllOfMP3

[UPDATE: It's not that simple. Apparently, although they are legal strictly speaking, they don't seem to be very fair. Bottom line is - there is a very big chance that the artist you're downloading songs of will not see a dime from it.
I was a little skeptical, so I started Googling with "AllOfMp3 scam". Take a look at one of the results... Look also at this...
I personally intend to liquidate the remaining 10$ or so I have left in my account and I doubt I will renew it. I really don't like the way the music companies extort us with their prices, but that doesn't mean I want to steal the music. I think that a more reasonable pricing should be available. I know some companies have started with new approaches (such as Napster), but they all sell protected files, which is not practical to me.

So I'm back to square one - looking for a good, reliable, legal, not too expensive (and fair) source of digitally available music...]

I’ve discovered this amazing site yesterday. It’s a legal songs download site, and it’s awesome!!! Their pricing makes so much more sense than all the other sites. They charge you by the byte – the larger the file you download, the more you pay. In addition, for most songs you can choose the exact encoding parameters: encoder, bitrate, other encoder-specific parameters. In the end, you end up paying much much less for your songs. You can roughly count on 10 cent/song with 192kbps encoding. Ah, I almost forgot, the download rate I get is over 160KB/s on average.

Did I mention I liked it, A LOT ?!?!

[Update: One thing I think they should improve is their search capabilities. Currently you can only search by artist, album or song. With such an extensive amount of data, I would expect much more flexible and advanced searching options. Hope this will improve over time...]

AIBO and QRIO - May you rest in peace...

For those who haven’t had the opportunity to meet AIBO or QRIO, I suggest you run looking at the video clips on Sony’s site, because Sony announced recently that they are stopping production of both.

I have had the opportunity to play a bit with AIBO, when Ehud Sharlin came to give a talk at our CS Seminar. This little robot-dog was really impressive. Put some fur on it, and you might really get attached to it! Being a dog-lover, I can’t say it’s the same as the real thing. However, being an AI fan, I must say it’s really impressive and a real loss for the AI community.

XOR operator in C#

There is a XOR operator in C#!!!
For some reason, it’s impossible to find it by searching for “XOR” in MSDN (either in the index or using the search). I don’t know why they’re trying to hide that…
So anyway, it’s a simple bitwise operator: ^
So:

C = A XOR B should be written: C = A^B;

(and there’s of course the complementary ^= which also performs an assignment)

Bringing your PC to a technician...

YNET published today a very interesting and sad article.

They took completely good computers, incorporated one technical problem to each of them and brought them to 6 different PC technical labs.
If you want the full details – have a go at the article on their site.

For those who want the summary (or don’t read Hebrew):

First technical problem: graphics card not right in its slot.
Results: 2 labs fixed the problem with no cost. 1 lab replaced the graphics card with a new one and demanded approx. 30$ for it.

Second technical problem: some of the memory was replaced with corrupted memory, which caused the computer to fail startup.
Results: 2 labs claimed the motherboard AND CPU needed to be replaced. 1 lab simply recommended upgrading to a completely new computer.

Need I say more?

Matlab number formatting

This is an issue that has annoyed many (me included).
Matlab, by default, displays floating point numbers in their ‘short’ format. That is, the number ‘0.181096612595921’ will be displayed in the Command Window as ‘0.1811’. The problem is that you have no indication whether that’s the whole number, or like in this case, only a rounded ‘short’ version of it. Also, it’s not very obvious how this should be changed in the environment (the Matlab Help is far from being the best in town).

So, for those of you who are interested, here it is:

1. Changing the format in using the Preferences (these changes will remain forever, until you change them again):
1. Changing the Command Window display:

Click on the menu: File è Preferences. Choose ‘Command Window’ and change the ‘Numeric Format’ combo.

2. Changing the Array Editor:

Click on the menu: File è Preferences. Choose ‘Array Editor’ and change the ‘Numeric Format’ combo.

2. Changing the format for the current session only: use the ‘format’ function. For example:

‘format long g’ – will display the number above as ‘0.181096612595921’,

Whereas ‘format short g’ will display it as ‘0.1811’.

For details about the various options, check the help on the ‘format’ function.

Wireless router

That’s it, I’ve had it! I’ve been suffering from my Wireless Broadband Router for the past half year and enough is enough!
When I bought my laptop last summer, I also bought a wireless router to go with it, of course. Not knowing what router’s best, I let the guy who sold me the computer to choose the router. So I ended up with a router by Edimax (BR6114WG to be accurate).
At first, everything went smoothly. The installation was fairly easy and I was relatively happy. For the first time in my live I had the possibility to work from my back yard, so I was ready to close an eye on the disconnections I experienced once in a while.
Then, with time, I encountered several problems:

1. The wireless network is really unreliable. Even from short distances, with little disturbance, the connection gets disconnected once in a while.
2. If the router remains on for several days (typically around 3 days), it suddenly stops providing Internet connection.
3. On two occasions the router simply dropped dead. The first one required a complete replacement (it took them 2 weeks to reach to this conclusion). On the second occasion, it suddenly came back to life after a few hours.
4. Once in a while, the router becomes ‘almost dead’. I won’t get into details, but just day that to revive it I had to unplug/plug it to the electricity several times in a row.

In the end, I decided that I’ve been wasting way too much time on this stupid piece of plastic. I checked out the user reviews in Zap and decided to buy a new 3COM router with 3Yrs guarantee. It’s 20-30% more expensive, but if it will save me the administration time, it’s certainly worth it.

I received it today. The installation was slightly more lengthy than with the Edimax (although the documentation is much more professional). I’m happy to say that it’s now up and running. I’ve got a good feeling about this – let’s hope I’m not wrong!

French Onion Soup

Yet another soup I really like for cold winter days. It’s extremely simple, and if you give it enough time it’s wonderful!
Last time I made it I used leek (‘kresha’ in Hebrew) instead of onions and it turned out to be maybe even better.

Ingredients:

• 4-8 Onions depending on their size. Could be replaced by the same quantity of leeks
• 50gr. Yellow butter
• 2 soup spoons of white wheat
• 1 liter of Chicken Bouillon
• 1/2 cup white wine
• 1/4 cup Brandy
• Salt, peper, Chili powder

Directions:

• Cut the onions in stripes. I simply cut them in half and then cut each half in thin slices.
• Fry the onions in the butter, using a high cooking pan. Make sure you keep the fire LOW! It should fry for around an hour, maybe even more, and still not burn. Keep stirring once in a while. You must continue frying until the onions are almost completely melted, transparent, with a slight yellow tint (not burned!). Most of the time I leave it on the fire with closed top (except for when I stir, obviously…).
• Add the wheat and stir thoroughly. Let them fry with the wheat for 1-2 minutes. You must keep stirring otherwise the wheat will burn.
• Add the bouillon and the wine. I usually warm the bouillon up beforehand, but it’s not a must. If you don’t have enough liquid to cover all the onions, you can complement with warm water (or more bouillon). Turn the fire up until it boils.
• Lower the fire and keep it at boiling temperature for another ½ hour (sometimes a little more). Make sure you continue to stir once in a while; otherwise the onions get stuck on the bottom and… burn.
• Add the Chili powder, salt and pepper – according to your taste
• Stir and leave on the fire for a couple of minutes
• Take off the fire and add the brandy, stir
• Mission accomplished

Side note: Chili powder is not something you’ll see often in an onion soup. I tried it once, and found it adds a special taste to the soup, makes it interesting. It also gives it a nice orange-red color (that is not due to over-burning the onions). I’ve never made an onion soup without it since then!

Building a DAL using Strongly Typed TableAdapters and DataTables - by Scott Guthrie

Scott Guthrie (a.k.a. ScottGu) is starting a series of tutorial postings on common data binding design patterns. He’s definitely done a great start with his first post about Building a DAL using Strongly Typed TableAdapters and DataTables.

Thanks Scott!

Mother of all soups - the Chicken Bouillon and a Bouillon-based vegetable soup

Almost all the soups I do are best made with a basis of a chicken bouillon. In some cases, you could do with just water, but usually the bouillon changes the soup from ‘good’ to ‘excellent’. I never use flavoring powders and the like. They are full of mono-sodium glutamate, which is unhealthy and sometimes difficult to digest…

Clear Chicken Bouillon

Ingredients:

• ~2 kg chicken parts. I usually prefer the neck. If I can’t find necks I use wings. Chicken comes out much better than turkey, in my opinion.
• 1 large celery root
• 1 or 2 parsley roots
• Various herbs for a “bouquet garnis”. I use mostly parsley and celery but feel free to use also coriander, dill or whatever you like.
• (optional) 1 or 2 small dried hot peppers
• 4-5 bay-tree leaves (“Laurier” in French or “Dafna” in Hebrew or “Laurus nobilis” in Latin)

Directions:

• Put all the ingredients in a large, high cooking pan.
• Fill with water such that all the ingredients are totally covered.
• Let it boil. Beware – the content has a tendency to overflow when the fire is too high. If it does overflow, it becomes really messy, so once it reaches boiling temperature, make sure to lower the heat.
• Leave it at boiling temperature for 2-3 hours (more towards 3..). Stir once in a while and make sure everything is covered by water. During the whole cooking, leave the cover almost closed (i.e. leave a small gap for the superfluous steam to get out). Some notes should be made at this point:
• If you leave the cover completely open, the soup will dry out to quickly and you’ll be left with too few liquid.
• If you leave the cover completely closed, there’s a good chance it will overflow. Don’t blame me at that point!
• If you stop cooking too early, the soup won’t have enough tastes.
• When it’s ready, transfer it to a large container, preferably one that is transparent and has a thin bottom (explanations why in a second). Make sure to filter out everything: the chicken, the herbs and especially the hot peppers. I usually give the chicken to my dog and dispose of all the rest.
• You may have some remaining residue in the new container – don’t worry about it for the moment.
• Leave it in the refrigerator for several hours (I leave it for the night).
• In the morning, what you should have in your container is a somewhat jelly-like substance. On the bottom you should see the entire residue, and on the top there is a layer of pure fat. Sometimes the fat can become creamy of even crusty; sometimes it’s much like heavy oil.
• The idea is to keep the jelly and drop the rest (fat and residue). Since the bouillon has become jelly, it’s a really easy job:
• With a spoon you scratch all the fat and dispose of it.
• Then you can extract the bouillon while leaving the residue in the container.
• That’s it! Now you have a strong, clear and rather low in fat bouillon. You could either keep it in small containers in the freezer for later use (either in soups or other dishes) or make some really good soup out of it right away!!

Very Simple Vegetable (but not Vegetarian) Soup

Ingredients:

• Strong, fresh, chicken bouillon
• Veggies J . I usually use the following:
• 3-4 pealed, sliced carrots
• 2 leeks (or 2 onions)
• Pumpkin (in cubes)
• 2-3 cloves of garlic, cut into slices
• (optional) half a cauliflower (“Kruvit” in Hebrew), in small pieces

Directions:

• Just put everything in a large cooking pan. Make sure there is enough bouillon to cover all the vegetables. Let it boil and leave it at boiling temperature for about 1 hour. Don’t let it overflow!
• When it’s ready, you can add spices. I would suggest only some salt. Since here in Israel the chicken comes already salted due to the koshering process, I actually don’t add anything.

Twist:

• If you want to give it an oriental taste, you can add the following spices (all or part).:
• Coriander (I’d add it to the bouillon)
• Cumin
• Turmeric
• A tiny bit of Cardamom (“Hel” in Hebrew)
• 2-3 Cloves (“Ziporen” in Hebrew) – I’d add it to the bouillon as well

Well, I think that’s it for now. The next soups (not necessarily in that order) will be: “French Onion Soup”, “Rich Meat and Lentil Soup” and maybe “Simple Pea Soup”.

Bon Appetit J

Soups

One of the few things I really like about winter is the soups. So since the winter is now really showing, here in Israel, I’ve decided to start posting a few messages with the recipes of the soups I like (and prepare) the most. If you happen to try preparing them, I'd be happy to hear how it went (for the good and for the bad).

Nasty SQL Code

Oren Eini (or is his name Ayende Rahien? I'm a little confused about that) has a very nice blog I like to read. He must be going through a really hard time, trying to deal with extremely crappy code lately. I suggest you take a look at his 3 pieces of nasty SQL code (here, here and here). It’s worth it!

I'm back to IE

A few weeks ago I posted that I’ve decided to try using FireFox, with the idea of opening anything that is not sufficiently supported by FireFox with the IETab. Well, I’m sorry to day that this experience didn’t prove successful for me, and I changed my default browser back to IE.
Just to name a few reasons:
1. IETab Pop-up blocking – there is no way (that I could find) to define on which sites you want to allow pop-up blocking and which not when you’re using the IETab. You must set pop-up blocking to either ON or OFF for ALL the sites being viewed with the IETab. This is the most significant reason why I disliked the IETab. As Murphy would have wanted it, most of the sites I need to view with the IETab (mostly but not exclusively bank account sites) use pop-ups. So I do need to allow pop-ups, but I certainly don’t want to allow them for all the sites viewed with the IETab!
2. Performance – odd isn’t it? Well, some sites take up quite a lot of CPU with all their flash publicities and all. I don’t like it, but since I like the content I live with it. One of these sites is YNET. The problem with FireFox, is that even when the browser is minimized, it takes up a significant amount of CPU when these sites are open. With IE, once the browser is minimized, the CPU drops to normal and is free for my other stuff.
3. Defining sites that should be displayed with IETab – the list is a list of exact urls. You can’t define web sites. This has proven to be very annoying.
4. I don’t remember exact examples at the moment, but I did encounter some sites that didn’t look well either with the IETab or without it.

Until these issues are not fixed, I’m sorry to say I’ll have to continue using IE. I know that I could use both, and each time choose the best one, but I hate working that way.

(Right before sending this post I discovered there is a new IETab release. I'll try and play with it some time soon. Maybe it will make me change my mind?)

Network resource KILLERS

I’m not a Web programmer, so I can’t say how you should write web sites. I do, however, have enough experience in software development and networking in order to know some of the thing you should not do.
I had to play quite a lot with the Windows Event Viewer lately. While working, I noticed an ever recurring warning that goes like so:
“TCP/IP has reached the security limit imposed on the number of concurrent TCP connect attempts.”
Since it even happened very shortly after having rebooted my system, I decided to track down the source of it. Here are results of playing with the netstat utility:

1. No particular application open. Netstat returns 3 TCP connection owned by the following processes: MsnMsngr.exe, Skype.exe and gnotify.exe. Makes sense, doesn’t it

2. I opened one instance of Internet Explorer and entered YNET. Now, in addition to the above-mentioned 3 connections I had 61 other connections, all owned by IEXPLORE.EXE

That’s definitely a huge WTF ?!?!?! You open one single web site and you’re swamped with so many connections that in the end the OS has to start imposing limitations because it believes you may be running a virus/worm or some other threatening application.
Needless to say that when I checked with FireFox the behavior was more or less the same (didn’t count exactly the number of connections but it’s close enough).
Also, most other sites I checked, although they do use quite a lot of connections, it’s still not that much.