I’ve been doing a lot of performance tuning the last few days using the NH profiler. We use Linq2NHibernate next-to-exclusive – so, I know, we could have solved the problems, I’m about to describe, using all the other query-mechanisms in NHibernate.
First off, there are some NotSupportedExceptions still in the Linq-implementation – e.g. you have to be careful where you put .Select statements (I found that if you put them before your .Skip, .Take but after your .OrderBy clauses – you should be okay.). These can be worked around, so they are nuisances mostly.
Next are the shortcomings – if you do paging be well aware of eager-loaded child collections. If you are not careful, you will be selecting the entire tables and doing sorting in-memory! Consider the following code:
var orders = session.Query<Order>().Take(20);
Since we are doing paging – we are probably aware of cartesian products, so we have placed a hint in the mapping that orderlines should be sent in a seondary SELECT (fetch=”subselect”), but if you notice the SQL that is sent to the DB – you will notice that although the order query is limited to a TOP 20 in the SELECT – the OrderLines are not. The biggest problem is that you will probably not notice this (unless you actively look at the SQL NHibernate is emitting) until you are in production since it isn’t really a problem in small result-sets, but when you get to having 100k OrderLines – you are bound for disaster.
So, since waiting for the next release of NHibernate is probably not an option, how do we work around this issue? Well, the easiest and most performant solution is to somehow include the TOP clause in the WHERE clause (since that is being carried into the sub SELECT). What you do is simply to do the query twice – first selecting only the Id’s (with all your WHERE clauses) and then querying for Orders matching these Id’s. This will result in the expected result with a nice IN clause in the sub SELECTS). Also, if your are doing arbitrary WHERE clauses on non-indexed columns – you will only query the Order table once on these – instead of doing it for all the sub SELECT’s as well.
So, I’ve been writing pretty much since Christmas on and off. I pretty quickly switched to LaTeX instead of Open Office. I downloaded the ‘book’-template and just copied everything in from the old document and have not looked back since. It is a joy writing as I would code – no fuss, no irritation, just flat-out writing and using tags to signify what is the header, what is a chapter heading – and everything is generated for me. It just goes to show that for professional stuff, wysiwyg just isn’t as important as stability and responsiveness.
So far, I’ve chosen to go the route of: make the code for the chapter and then writing it afterwards and then take the next chapter the same way. It is a fun project and having the first 9 chapters under my belt feels good. So far, I’ve been far about introducing a lot of patterns, principles, Prism, NHibernate and WCF! It has been a joy to write as it kinda writes itself. I hope that at some point, I will be able to get the opinions of my peers to see if my work is actually readable and makes sense to anyone but me.
9 chapters is about 137 pages in the book. This is a bit of cheating though, because I rely heavily on code examples and since I have done nothing about formatting it, it takes up more space than it probably will once I get the formatting down.
Having threatened to do this for a number of years – I’ve finally decided to give it a go. It will be fun to see how it turns out – so far the idea is to write a book about how I develop software. It will be very hands-on and targeted at experienced developers. The sole reason for the book is that I haven’t been able to find one with the content I want in a programming book. I will be writing an application during the book – complete with the technology stack that I’m usually using (WPF, WCF, NHibernate, Prism (incl. Unity and MEF). The thing I’ll be doing differently is that I’ll be re-factoring the application throughout the book as it progresses as features of the application emerge.
So, you’ve probably heard the thing about doing a book like none out there before… The idea here is not to show-case the technology, but rather show how I write software. What are my thoughts (if any…) during the process – how do I start and last but not least – how do I keep the application from turning into a maintenance nightmare? The key is adhering to SOLID as much as is feasible. I’ve decided to not use TDD-methodologies per se, but rather facilitate testing by using interfaces and Dependency injection. I’ll post more – when I’ve actually gotten something down on paper… erh… in Open Office.
This post originates from this StackOverflow question.
At work, we’re working on a solution which should include some kind of version-control document-store to be shared between users. As some of our customers are not that skilled with computers, we had a few extra requirements:
- Should be seamless worked into the application
- Should auto-commit when a document is edited
We threw a few ideas around – everything from using SharePoint through a simple web-archive with ftp-commands getting the files to-and-from the server. And then one of my colleagues asked – “Why not just use SVN?”. And yeah, why not use something well-tested and something that we were already using… So, I turned to StackOverflow to see if anyone had tried something similar. The responses were varied, but seemed to settle in two camps – the “Go for SharePoint”-one and then “Should work… but you should try out Mercurial/Git instead”. SharePoint isn’t really an option for us as it the cost is prohibitive for our use (And I’ve had really bad experiences with upgrading between versions).
I decided to go with SVN for our prototype (which is now integrated and working albeit with limited functionality) as that was the easiest option to see if it was at all feasible. We use the SVNSharp library for interacting with the repository – although it has not been upgraded from .Net 2.0 it seems pretty stable (aside from a few quirks…). I’ll outline the functionality we have working along with our findings below:
Authentication
This had me stumped for a while. I could easily get it working on my own developement-station, but that has the Tortoise SVN client installed, so it would use that cache no matter what I told the client to use… So, I had to do a bit of trial and error before figuring this one out. Basically, you need to tell the client to use a specific user and password (and domain if the client is outside the domain) as the default authentication. So that information went into the customer table.
Viewing documents and repository structure
This was by far the easiest part. We decided on using a repository per customer (around 4 users on average) – we need more info on performance, risks of corruption before turning to a better solution here. We need to make sure that two customers cannot see each other’s documents, and this seemed the easiest solution (at least for the prototype, this is a pretty good solution). Basically every customer has a row in a specific table in the database and it was a simple matter of adding the URL for the repository.
Adding a file to the repository
This took a bit of fiddling – basically, I ended up with exploiting the CheckOut call of the client which has a depth-parameter – then I organized the document so they are split across a number of folders (according to which record they belong) and then checked out only the directories. If a record doesn’t have a directory, it is added and checked in. Then I copy the document into the folder and check that in. Pretty basic stuff untill the application started crashing on me. It turns out that you must supply a CommitArgs when commiting – otherwise it will crash with no error-message at all… That wasn’t really obvious, but it might be due to our specific setup – haven’t fiddled much with it – and I should supply commit-args anyway as that is where the commit-notes go.
Checking out a file not in the local copy
This one was a bit tricky as SVN does not directly support checkout of a single file, and I really didn’t want to download the whole folder every time I needed one document. It turns out that if instead of doing a CheckOut – you can use the Update-statement instead (which supports single files) – you just need to know the exact path in the repository (And since I read the contents from the repository instead of the local copy – this is actually a pretty nice solution).
Autocommitting
This one took a bit of thought, and the solution we have now isn’t exactly perfect. I ended up with hooking into the process that I start when launching the file (I use Process.Start) – and then subscribing to the Exited-event (and remembering to EnableEventsToBeFired (or some such)), so it would actually fire. This has a few draw-backs – if the client is exited (or crashes), I loose any connection to the open application(s) and cannot determine which documents were altered. I could do a full commit, but I don’t know when the client will be restarted (and if). Also, the document must be opened from the application to notify correctly. I’ll probably develop a small service-client that will monitor the filesystem instead for the final version.
Finishing touches
I really wanted to have small icons on the files to show which application would open the document – so, I found a solution online. Appearantly, there is no way to hook into the registry for the icons using C#, so, the solution uses a lot of HWND-calls. It works, but I have no idea how :).
So, it looks really promising – I’ll probably look into Git/Mercurial to see if I can get a cleaner approach, but so far just the SVN-client is really nice. It has a few quirks yet which need to be ironed out (such as when you add two files too quickly after each other, the application will crash horribly).
Or how to create a minimalistic chat-client using WPF and WCF in 75 lines of code (according to VS). What I want to show with this example application is the bare minimum for achieving cross-client notifications (or at least one way of doing it). For those of you who want to jump right ahead to the code – go here.
The scenario
Basic chat – one client publishes a message – all clients adds this message to their message history.
I won’t go into details on how the UI is built – it’s minimalistic and isn’t really important for what I wish to show. Let’s instead take a look at the service implementation:
[ServiceContract]
public interface IChatRelayService
{
[OperationContract]
string[] GetMessagesSince(int value);
[OperationContract]
int GetLastMessageId();
[OperationContract]
void PublishMessage(string message);
}
So, a couple of questions would probably arise here. If I’m keeping it simple, why three methods – I could have just two… downloading all the messages instead of all the keeping track business. There are (at least) a few major good reason for this:
- Performance – I want to keep the “Are there any messages for me?”-call as lightweight as possible. That way one service will be able to service a lot of clients. Downloading the messages can then be handled by another (possibly clustered) service.
- Simplicity – I want to offload as much as possible about keeping track to the client. On the client I will have to keep track of that one client. The Service is keeping track of a lot of clients and should have as little processing work as possible.
- Multi-threading – the service will be pestered with calls from clients, so we will have contention for shared resources. I will reduce the amount of housekeeping code drastically by having as little state as possible on the server.
Right, the implementation of the service is straight-forward – so look in the accompanying code to see that. What we’re really interested in is the client. I’ve come up with the following requirements:
- It should check for messages every 500 ms.
- It shouldn’t read messages posted before starting
- It should be lightning-fast (no blocking of the UI-thread)
Alright, I’ve determined I need four classes:
- A MainWindow with a TextBox for entering messages, a Button to send it and a ListBox to show all messages.
- A ChatListener for communication with the ChatRelayService and updating when there are new messages.
- A ChatViewModel that will keep state for the MainWindow and will handle interaction with the ChatListener
- A SendMessageCommand for the Button in MainWindow.
The ChatListener
This is the most interesting class in this little app. Let’s look at the code first – then discuss it:
public class ChatListener
{
private readonly ChatRelayServiceClient _client = new ChatRelayServiceClient(“ChatRelayService”);
private readonly BackgroundWorker _worker = new BackgroundWorker();
private int _lastMessageId;
public ChatListener()
{
_worker.DoWork += ListenForNewMessages;
_worker.RunWorkerCompleted += FoundUpdates;
_lastMessageId = _client.GetLastMessageId();
StartWorker();
}
public event Action> NewMessages;
public void BroadcastMessage(string message)
{
_client.PublishMessage(message);
}
private void StartWorker()
{
_worker.RunWorkerAsync(_lastMessageId);
}
private void FoundUpdates(object sender, RunWorkerCompletedEventArgs e)
{
var newMessages = _client.GetMessagesSince(_lastMessageId);
_lastMessageId += newMessages.Length;
if (NewMessages != null)
NewMessages(newMessages);
StartWorker();
}
private void ListenForNewMessages(object sender, DoWorkEventArgs e)
{
var lastEventId = (int) e.Argument;
do
{
Thread.Sleep(500);
} while (_client.GetLastMessageId() == lastEventId);
}
}
Okay, a few notes: I’m relying on the BackgroundWorker for getting the async-feeling right away without messing (too) much with Threading. Basically, I start out in the constructor with setting up the BackgroundWorker, getting the lastMessageId (since I don’t want all the old messages) and then starting the worker. The worker then checks every 500 ms if there is a new highest MessageId – if that’s the case, it returns to the main thread, downloads the messages and notifies everyone interested in the new messages, and starts listening for new updates – rinse, repeat.
I won’t go into details with the rest of the classes as the ChatViewModel simply listens for messages from the ChatListener and updates an ObservableCollection when it has new messages.
So, that’s it for a simple chat-client. There are a number of issues with the current code, but it shows the basic ideas for a loosely coupled, scalable way of keeping clients in sync. In a real-world application – a database would probably be involved serverside, the messages would probably contain enough information for the client to update correctly and finally, the notifications would be run in a separate service. One of the major benefits is that we are not relying on clients keeping an open connection to the webservice nor any duplex-bindings or equally complex. It’s very basic, very simple, it scales really, really well and most important – it is hard to break.
When you have multiple clients working on the same data – concurrency rears it’s ugly head. How to handle two users loading the same data – and both of them editing it and trying to save it back to the DB. There is no way to keep everyone happy when this happens – either Last-In wins (potentially over-writing First-Ins work or putting objects in invalid business state) or First-In wins and Last-In’s data is lost. Of course, there are people who would try to merge the changes (and good luck to them…), but it’s not my first choice. I would rather keep it from happening in the first place.
And then comes the funny part trying to find a bullet-proof way of doing that. DBA’s would come up with a clever locking-scheme that would lock the row when one user starts editing (ReadLock-kinda-way). This might work (and when I say might – probably would not work). The problem is that we re usually in a disconnected scenario – it’s just not feasible setting a read-lock for a client, you don’t know will return it promptly. What if the client is disconnected.. what if… and so on. Disconnected clients (for obvious reasons) cannot be counted on to behave like connected clients. Besides cleaning up dead-locks in a database is just never fun (and having a shared data service hanging indefinitely is really never fun).
So, read locks at the database level are out (well, it was never really in – I just mentioned it, so DBA’s wouldn’t be left out of the fun debate). The problem is that right now, we are trying to tackle the problem at the service level – we need to go back to why the problem is happening in the first place. The problem is users are not good at telling other users what they are doing – seldom do we have an office where a user will send an e-mail around to hear if anyone is editing this Account – waiting to get replies from everyone – then going to Peter who was busy playing Minesweeper and didn’t have time to answer mails and determining that he doesn’t have that particular account open – finally, coming back to his own computer – edit the account using the 10-digit system (thus being done in 2 seconds) and promptly saving the Account. This Does Not Happen.
Users don’t work that way… They’ll open the Account, gloss over it sipping their coffee, edit one property, decide that was wrong, delete the entire field, pick up the telephone to ensure that their cable is installed today, come back to the screen, stare blankly at the screen for several minutes trying to discover what they were doing. After asking their colleagues what the score is in the NBA play-offs, they will remember that they were editing the phone-field and re-enter the data in the fax-field they had erased and then hit the save button.
The grace period we are talking here – can be several hours because most users are not very good at working like a computer. They are a spontaneous crowd who make mistakes… constantly. And most will be juggling several things at once – e-mail, phone, fax and the occasional daily-yelling-from-the-angry-boss will be competing with your application for their attention (and guess who is winning… if you guessed your application, you need to see a psychiatrist about delusions of grandeur…).
So, what to do – well, I believe in using every available UI-widget at hand to signal to the user what is a good idea – and what is a not-so-good-idea (“Are you certain that you wish to delete the database, loosing three years of work” combined with flashing radioactive signs, Darth Vader popping up to call the user a moronic imbecile who was to blame for the death (pun intended) of the Death Star). Okay, you could make it less flashy, but nobody likes a boring screen. What I’d do was to show the user a ribbon at the top of the screen alerting him that Alex did some changes – and that if he’d like, the Account can be reloaded to reflect the changes. The reason, I wouldn’t just go ahead and do it – the user would be angry at me for not letting him jot down his changes before the screen flickered (the loss of work again). And if I was feeling particularly in a good mood that day – I would even add functionality to apply his changes after the reload (if possible).
This approach would almost eliminate the friction that users feel when working on distributed data. The one thing that we cannot deal with is users supplying data at the exact same second. But we would have backup procedures for that. (“Sorry, we were unable to save your changes as Alex beat you to it – here is his e-mail: alex@someplace.com if you feel like venting”). And guess what? (No, it is not a knock-knock joke) Your first and foremost responsibility as a developer is removing friction from users! If they feel friction – they will think you application is lame no matter how brilliant everything else is.
In my next instalment, I’ll talk about how to solve this technically.
I went ahead and got the Eric IDE to start a bit of a project with some UI involved. I found a nice tutorial for a parser.
Of course, it would be too easy to install it on Windows – so, I did it all on my netbook… first things first. Don’t do that for production… A 1024 * 512 resolution is too small – there are a few glitches with the IDE as the menus are a bit too large – so, it jumps around a bit, but I still managed to get through the tutorial. Coming out on the other side – I’m really impressed by Pythons capabilities for making clean, compact code – and having concluded the tutorial – I now know how to:
- Install a few missing packages (Needed a Qt Designer and a compiler for the xml that Qt Designer generates).
- Install and setup Subversion repositories and use that together with the IDE.
- Some small time UI-developement – of course only brushed the surface there.
And I actually understand the Python code. Comparing it with back when I started C# coding… well, it’s an unfair competition – it took me several weeks to get this far. The reason that it is unfair, is that I learned OO-programming whilst learning C#. Picking up the 4th language even though it is dynamic is a lot easier. It’s a lot of fun programming in Python – it’s basically learning a few rules (line indents matter, interfaces are meaningless) and then you’re off writing small programs.
Oh – and for those wondering, these are the languages, I know:
- VBA – Excel primarily back when MS Office was called 2000/2002
- XAL Basic – Scripting language in Microsofts Dynamics products
- C#
- Python
Of those, I would be hard-pressed to anything really usefull in VBA these days (and C64 BASIC, batch file scripting and VB-scripts didn’t make the list although I’ve done it, but I could probably only edit and read them now). I like learning new languages (and come September, I’ll start learning Spanish…)
When in Rome… After installing Linux – it was kinda anti-climatic. Everything was installed without me doing anything – and using the browser and the small utilities quickly was… meh. So, new project – learn a programming language that would run on Linux.
Enter a lot of confusion. I had no idea what-so-ever where to start. So, turning to StackOverflow (this was the question and responses) – I decided to try out Python – I mean how hard could it be… Installing Vim with the Package Manager was easy… and I couldn’t find it… I looked through all the menus… tried re-installing it… read up on it on google… Saw Cream (GUI extension on top of Vim) and installed that. Woo, an editor… erh… texteditor. I searched in vain for the Build and Run functions…
Then it dawned on me – Windows methodology doesn’t work on Linux. Back to the drawing board (read: Google). I had to save the edited file – and once it had a .py extension – I had syntax high-lighting (well, to some extent – a dynamic language doesn’t really have that much that highlights). In the process of reading up on how to program in Python – I did the ‘When in Rome…’-deal. I turned to the Terminal – and now things started to pick up. Two terminals – one running Vim – the other just being a terminal. Save in Vim – Execute on the Terminal.
I found a cheat-sheet on Vim – and it can really do magic. I haven’t found more than 10 shortcuts that I can remember – but the list is growing pretty quickly – and I already wish R# had these… And I like the ! when you mean business. :q means “Quit, but if i forget to save – warn me” :q! means “Quit, GDI”. I like having a GDI-command… (For those of you wondering: GDI – GodDammIt).
Now, to Python. Working with a dynamic language is really a shell-shocker when you are used to static type-checking. For me so far – it feels like the safety-net has been taken away… my training wheels are off, and my dad isn’t holding the pole… My parachute was left back in the cockpit… well, you get the picture. It feels like you are cheating (but.. but… I didn’t say it was a float… why do you treat it like one). And no debugger to say: “You forgot the ;, you moron” when you compile… because… you don’t compile. Python is compiled on the fly.
I’m still with the basic stuff – learning to load libraries and my own files. So, stay tuned for more Python madness.
I had to try…
Positive impression so far. Just made a simple console application. Everything is lightning fast – even if I’m sitting on a 0.99 GHz ATOM CPU, 1 GB Ram and a very small hard-drive – I was amazed that it would even start – but this speed? I wouldn’t have thought it possible. Of course – things might be another story once, I start adding third-party assemblies etc.
And this package-deal with Linux… I can’t help but wonder if it isn’t the way to go… 90 MB (and that was because, I clicked a few extras) to have the full MonoDevelop application installed… good luck trying that with Visual Studio :).
Of course – I wouldn’t like to do production code with MonoDevelop (not yet anyway), but I think it’s worthwhile to check out the competition from time to time.
I’ve been threatening to do this for some time – but now, I finally took the plunge. I have a little net-book that’s a perfect candidate for experiments. I started this afternoon and now a few hours (took me 6 hours before I was fully online), I’m reporting to you live from my new Linux Mint 9 (main)!
First things first – I had to choose a distro (my vocabulary has really expanded today). It had to be something light-weight – and after reading through reviews… trying to decipher which meant what… I closed my hand and pick the one, I hit first with the mouse… Then came the download, and while that was happily churning away – I read up on the user guide for Linux Mint. Turns out, the default is to install it on a DVD/CD… which is a no-go on this netbook…
With some trepidation – I google’d for way to circumvent this. First link – first win! There was a step-by-step tutorial… Download this program… select the downloaded distro (it could actually download it if I hadn’t already started that) – and point at the USB to serve as ‘Live’-USB.
Now, booting from the USB worked perfectly (and is a really nice feature by-the-way). I had read the user guide – so I went ahead and tried to install it on the harddrive – since boot-time was not that good from USB (Linux is fast… but not that fast). First problem – partitioning didn’t show all the possibilities (I wanted a side-by-side installation). Google to the rescue – turns out there is a bug if there are problems with the harddrive – reboot to windows and did a chkdsk – back to install. Now everything went perfect with the install – except for one little detail… WiFi…
Uhoh – I knew that is one of the major pain-points in Linux – WiFi and printers has historically been pains when installing Linux judging from the comments in various forums. So, I started approaching the problem like I would with Windows… looking for a driver download… this is not how you do it in Linux… Instead of using hours trying to find a workaround – I should have picked up that Ethernet cable and plugged directly into the wireless router -> Gone to ‘Hardware drivers’ -> Selected the proprietary driver -> clicked ‘Activate’… Okay, that was an eye-opener as to how far Linux has come on the desktop.
If I had been a little bit more informed – installing Linux Mint would have been a cinch and I would have been installed in under 2 hours including downloading for an hour. But still – I am nowhere near installed on a Windows box in 6 hours… Here, I had to restart once (going form USB install to Harddrive install) – and all my usual programs are here by default (sans Visual Studio for obvious reasons). Hello Firefox, VLC, OpenOffice… and things are really snappy with limited resources… And updating to latest versions on everything with one click… that’s just awesome!
So, first try would qualify for a near-total success! Will be fun to see how it goes with day-to-day using it.
« Previous Page — « Forrige indlæg« Previous Page · Next Page »Næste indlæg » — Next Page »