Monday, 14 April 2014

Roundtrip Serialization of DateTimes in C#

Say you need to serialize a DateTime object to string (perhaps for an XML or JSON file), and want to ensure that you get back exactly the same time you serialized. Well, you might think that you can just use DateTime.ToString and DateTime.Parse:

var testDate = DateTime.Now;
var serialized = testDate.ToString();
Console.WriteLine("Serialized: {0}",serialized);
var deserialized = DateTime.Parse(serialized);
Console.WriteLine("Deserialized: {0}",deserialized);

// prints:
Serialized: 14/04/2014 16:32:00
Deserialized: 14/04/2014 16:32:00

At first glance it seems to work, but actually this is a very bad idea. What’s wrong with this approach? Well two things. For starters, the millisecond component of the time has been lost. More seriously, you are entirely dependent on the deserializing system’s short date format being the same as the serializing system’s. For example, if you serialize in a UK culture with a short date format of dd/mm/yyyy and then deserialize in a US culture with a short date format of mm/dd/yyyy you can end up with completely the wrong date (e.g. 1st of March becomes the 3rd of January).

So how can you ensure you get exactly the same DateTime object back that you serialized? Well the answer is to use ToString with the “O” format specifier which is called the “roundtrip format specifier”. This will not only include the milliseconds, but also ensures that the DateTime.Kind property is represented in the string (under the hood a .NET DateTime is essentially a tick count plus a DateTimeKind).

Here’s three dates formatted with the roundtrip format specifier, but each with a different DateTime.Kind value:

Console.WriteLine("Local: {0}", DateTime.Now.ToString("O"));
Console.WriteLine("UTC: {0}", DateTime.UtcNow.ToString("O"));
var unspecified = new DateTime(2014,3,11,16,13,56,123,DateTimeKind.Unspecified);
Console.WriteLine("Unspecified: {0}", unspecified.ToString("O"));

// prints:
Local: 2014-04-14T16:36:01.5305961+01:00
UTC: 2014-04-14T15:36:01.5345961Z
Unspecified: 2014-03-11T16:13:56.1230000

As you can see, UTC times end in Z, local times include the current offset from UTC, and unspecified times have no suffix. You can also see that we have fractions of a second to 7 decimal places. If you’ve serialized your DateTime objects like this, you can then get them back to exactly what they were before by using DateTime.Parse, and passing in the DateTimeStyles.RoundTripKind argument:

DateTime.Parse("2014-03-11T16:14:27.8660648+01:00", CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind);

The only thing to note here is that the serialization of a local time includes the offset from UTC. Which means that if you deserialize it in a different timezone, you will get the correct local time for that timezone. Which will of course be a different local time to the one originally serialized This is almost certainly what you want, but it does mean that technically, the binary representation of the deserialized DateTime object is not identical to the one that was serialized (different number of ticks).

Wednesday, 9 April 2014

Thoughts on Windows RT Audio APIs and BUILD 2014

One of the sessions I was most excited about at BUILD this year was Jason Olson and Pete Brown’s session on Sequencers, Synthesizers, and Software, Oh My! Building Great Music Creation Apps for Windows Store.

I had been hoping before the conference that we might see MIDI support for WinRT...

...and I wasn't disappointed. The talk covered a brand new MIDI API for Windows RT (requires 8.1 Update 1), as well as giving some guidance on low latency WASAPI usage.

Let's look first at the MIDI support…

Basically, it allows you to access MIDI in and out devices (MIDI over USB), receiving and sending messages. This is great news, and enables the creation of several types of application not previously possible in Windows Store apps (I’d wanted to make a software synthesizer for quite some time, so that’s first on my list now).

It’s distributed as a NuGet package allowing access from C#. This is huge in my opinion. One of the most painful things about doing audio work in C# has been endlessly creating wrappers for COM based APIs such as WASAPI and Media Foundation. It's really hard to get right in C# and you can run into all kinds of nasty threading issues. I had hoped when WinRT originally came out that it would wrap WASAPI and MediaFoundation in a way that eliminated the need for all the interop in NAudio, but sadly that was not to be the case. But this represents a great step in the right direction, and I hope it becomes standard.

What can't it do:

It doesn't read or write MIDI files, but that's not a particularly big deal to me. NAudio and MIDI.NET both can do that.

It doesn't provide a MIDI sequencer. You'd need to schedule out your MIDI out events at exactly the right time to play back a MIDI file from your app.

It doesn't offer up a software synthesizer as a MIDI out. That would be a great addition, especially if you could also load the software synthesizer up with SoundFonts.

There is no RTP MIDI support but they suggested it might be a feature under consideration. This would be useful for wireless MIDI, and could also form the basis for sending MIDI streams between applications which could also be useful.

And now onto WASAPI

There was some really useful information in the session on how to set up your audio processing to work at a pro audio thread priority, and how to use a “raw mode” to avoid global software based effects (e.g. a graphic equaliser) introducing additional latency. Getting good information on WASAPI is hard to come by, so it was good to see recommendations as well on using event driven mode and when to use exclusive mode.

What I would love to see with WASAPI is exactly the same approach as has been taken with MIDI. Give us a NuGet package that presents both the Render and Capture devices in a way that can be used from C#. Let me easily open an output device, specify if I need pro audio thread priority, raw mode, exclusive mode, and then give me a callback to fill the output buffer on demand with new samples. Ideally it should incorporate resampling too, as it is a huge pain to resample yourself if you are playing a 44.1kHz file while your soundcard is operating at 48kHz.

Likewise with capture. Let me easily open any capture device (or loopback capture), and get a callback whenever the next audio buffer is received containing the raw sample data. This might be possible already with the MediaCapture class but I'm unclear how you get at the raw samples (somehow create a custom sink?). Again, I’d like to be able to specify my capture sample rate and have the API resample on the fly for me, as often with voice recordings, 16kHz is quite acceptable. WASAPI is much harder to use than the legacy waveIn APIs in this regard.

Now Jason Olson did provide a link to a cool sample application on GitHub, showing the use of these APIs in C++. However, there seems to be a general assumption that if you're using WASAPI you are going to be working in C++. This is probably true for pro audio music applications. Despite having invested a huge amount of my time over the last 12 years in creating a C# audio library, I still recommend to people needing the lowest possible latency that they should consider C++.

But I also know that there is a huge amount of audio related development that does not require super low latency, particularly when dealing with voice recordings such as telephony, radio or VOIP. Recording and replaying voice is something that many businesses want to create software for, and a C# API greatly simplifies the development process. Both my experience from supporting NAudio, and having worked a decade in the telecoms industry tells me that there is a huge market for applications that deal with speech in various formats.

Finally, I'd also like to see Microsoft give some thought to creating an audio buss that allows WinRT apps to transmit real-time audio between themselves, allowing you to link together virtual synthesizers with effects. This could form the basis of solving the audio plugin problem for DAW-like applications in the Windows store.

Anyway, they repeatedly asked for feedback during the talk, so that’s the direction I’d like to see WinRT audio APIs going in. Seamless support for using audio capture and render devices from C# with transparent resampling on playback and record.

Wednesday, 2 April 2014

Stop Making Monoliths

Martin Fowler and James Lewis recently published an excellent article on microservices, which describes a new approach to architecting large enterprise systems, by composing them out of many independently deployable services, rather than producing a "monolith", which is usually one giant application that must be deployed/installed in one hit, and usually runs as a single process.

What's wrong with monoliths?

The trouble with monoliths is they become increasingly complex and unwieldy as the project grows larger. Despite our best intentions they usually become very tightly coupled, meaning components cannot be replaced or reused. Performance also becomes an issue, and there is no easy way to scale this approach other than to buy a more powerful server.

Why do we build them?

So why do we keep making monoliths? Two reasons. The first is simply because we can. I wrote a while ago about the fact that the vast increase in computing power has enabled us to build applications that are bigger than they ought to be.

The second reason is because it is the path of least resistance. It's far easier to put new code into the existing service, than to create a new service, which will require us to set up a new build and deployment process. So with a microservice architecture, you have to make it really easy to create a new service, or developers will end up bloating the existing services with additional features. Once the friction of creating new services is eliminated, you can begin to migrate away from your monolithic architecture.

What are the benefits of microservices over monoliths?

What benefits do microservices give over a monolithic architecture? Let me briefly mention two.

First microservices gives us real decoupling. Components in microservice architectures are actually replaceable rather than theoretically replaceable. The service boundary forces a radical decoupling of parts of our application, that is hard to enforce in a monolithic codebase, as it is too easy to circumvent (deliberately or unintentionally).

Second, it gives freedom to adopt new technology. Monoliths typically have to be built with one programming language, one database access technology, one dependency injection framework etc. It makes it easy to get stuck on legacy technology. But a microservice architecture allows you to incrementally improve your application to make use of better technologies, tools and practices.

There are plenty more benefits (not least the implications for scalability). But I'm hoping to blog more about other aspects of microservices, so I'll save them for next time.

Friday, 28 March 2014

How to record and play audio at the same time with NAudio

Quite often I get questions from people who would like to play audio that they are receiving over the network, or recording from the microphone, but also want to save that audio to WAV at the same time. This is actually quite easy to achieve, so long as you think in terms of a “signal chain” (which is something I talk about a lot in my audio courses on Pluralsight).

Basically, the usual strategy I recommend for playing audio that you receive over the network or from the microphone is to put it into a BufferedWaveProvider. You fill it with (PCM) audio as it becomes available, and then in its Read method, it returns the audio, or silence if the buffer is empty.

Normally, you’d pass the BufferedWaveProvider directly to the IWavePlayer device (such as WaveOut, but to implement save to WAV, we’ll first wrap it in a new signal chain component that we’ll create for this purpose. We’ll call it “SavingWaveProvider” and it will implement IWaveProvider. In it’s Read method, it will read from it’s source wave provider (the BufferedWaveProvider) in our case, and write to a WAV file before we pass it on.

We’ll dispose the WaveFileWriter if we read 0 bytes from the source wave provider, which should normally indicate we have reached the end of playback. But we also make the whole class Disposable, since BufferedWaveProvider is set up to always return the number of bytes we asked for in Read, so it will never reach the end itself.

Here’s the code for SavingWaveProvider:

class SavingWaveProvider : IWaveProvider, IDisposable
    private readonly IWaveProvider sourceWaveProvider;
    private readonly WaveFileWriter writer;
    private bool isWriterDisposed;

    public SavingWaveProvider(IWaveProvider sourceWaveProvider, string wavFilePath)
        this.sourceWaveProvider = sourceWaveProvider;
        writer = new WaveFileWriter(wavFilePath, sourceWaveProvider.WaveFormat);

    public int Read(byte[] buffer, int offset, int count)
        var read = sourceWaveProvider.Read(buffer, offset, count);
        if (count > 0 && !isWriterDisposed)
            writer.Write(buffer, offset, read);
        if (count == 0)
            Dispose(); // auto-dispose in case users forget
        return read;

    public WaveFormat WaveFormat { get { return sourceWaveProvider.WaveFormat; } }

    public void Dispose()
        if (!isWriterDisposed)
            isWriterDisposed = true;

And here’s how you use it to both play and save audio at the same time (note this is very simplified WPF app with two buttons and no checks that you don’t press Start twice in a row etc). The key is that we pass the BufferedWaveProvider into the constructor of the SavingWaveProvider, and then pass that to waveOut.Init. Then all we need to do is make sure we dispose SavingWaveProvider so that the WAV file header gets written correctly:

public partial class MainWindow : Window
    private WaveIn recorder;
    private BufferedWaveProvider bufferedWaveProvider;
    private SavingWaveProvider savingWaveProvider;
    private WaveOut player;

    public MainWindow()

    private void OnStartRecordingClick(object sender, RoutedEventArgs e)
        // set up the recorder
        recorder = new WaveIn();
        recorder.DataAvailable += RecorderOnDataAvailable;

        // set up our signal chain
        bufferedWaveProvider = new BufferedWaveProvider(recorder.WaveFormat);
        savingWaveProvider = new SavingWaveProvider(bufferedWaveProvider, "temp.wav");

        // set up playback
        player = new WaveOut();

        // begin playback & record

    private void RecorderOnDataAvailable(object sender, WaveInEventArgs waveInEventArgs)
        bufferedWaveProvider.AddSamples(waveInEventArgs.Buffer,0, waveInEventArgs.BytesRecorded);

    private void OnStopRecordingClick(object sender, RoutedEventArgs e)
        // stop recording
        // stop playback
        // finalise the WAV file

This technique isn’t only for saving audio that you record or receive over the network. It’s also a great way to get a copy of the audio you just played, which is very handy when you want to troubleshoot audio issues and want to get a copy of the exact audio that was sent to the soundcard. You can even insert many of these at different places in your signal chain, to hear what the audio sounded like earlier in the signal chain.

Friday, 14 March 2014

Detecting Mouse Hover Over ListBox Items in WPF

I’m a big fan of using MVVM in WPF and for the most part it works great, but still I find myself getting frustrated from time to time that data binding tasks that ought to be easy seem to require tremendous feats of ingenuity as well as encyclopaedic knowledge of the somewhat arcane WPF data binding syntax. However, my experience is that you can usually achieve whatever you need to if you create an attached behaviour.

In this instance, I wanted to detect when the mouse was hovered over an item in a ListBox, so that my ViewModel could perform some custom actions. The ListBox was of course bound to items in a ViewModel, and the items had their own custom template. You may know that unfortunately you can’t bind a method on your ViewModel to an event handler, or this would be straightforward.

My solution was to create an attached behaviour that listened to both the MouseEnter and MouseLeave events for the top level element in my ListBoxItem template. Both will call the Execute method of the ICommand you are bound to. When the mouse enters the ListBoxItem, it passes true as the parameter, and when it exits, it passes false. Here’s the attached behaviour:

public static class MouseOverHelpers
    public static readonly DependencyProperty MouseOverCommand =
        DependencyProperty.RegisterAttached("MouseOverCommand", typeof(ICommand), typeof(MouseOverHelpers),
                                                                new PropertyMetadata(null, PropertyChangedCallback));

    private static void PropertyChangedCallback(DependencyObject dependencyObject, DependencyPropertyChangedEventArgs args)
        var ui = dependencyObject as UIElement;
        if (ui == null) return;

        if (args.OldValue != null)
            ui.RemoveHandler(UIElement.MouseLeaveEvent, new RoutedEventHandler(MouseLeave));
            ui.RemoveHandler(UIElement.MouseEnterEvent, new RoutedEventHandler(MouseEnter));

        if (args.NewValue != null)
            ui.AddHandler(UIElement.MouseLeaveEvent, new RoutedEventHandler(MouseLeave));
            ui.AddHandler(UIElement.MouseEnterEvent, new RoutedEventHandler(MouseEnter));

    private static void ExecuteCommand(object sender, bool parameter)
        var dp = sender as DependencyObject;
        if (dp == null) return;

        var command = dp.GetValue(MouseOverCommand) as ICommand;
        if (command == null) return;

        if (command.CanExecute(parameter))

    private static void MouseEnter(object sender, RoutedEventArgs e)
        ExecuteCommand(sender, true);

    private static void MouseLeave(object sender, RoutedEventArgs e)
        ExecuteCommand(sender, false);

    public static void SetMouseOverCommand(DependencyObject o, ICommand value)
        o.SetValue(MouseOverCommand, value);

    public static ICommand GetMouseOverCommand(DependencyObject o)
        return o.GetValue(MouseOverCommand) as ICommand;

And here’s how you would make use of it in a ListBoxItem template:

<ControlTemplate TargetType="ListBoxItem">
    <Border Name="Border"
           my:MouseOverHelpers.MouseOverCommand="{Binding MouseOverCommand}">
        <Image Source="{Binding ImageSource}" Width="32" Height="32" Margin="2,0,2,0"/>

This is essentially a specialised version of the generic approach described here for binding an ICommand to any event. My version simply saves you needing a separate command for MouseEnter and MouseLeave.

Friday, 7 March 2014

Python Equivalents of LINQ Methods

In my last post, I looked at how Python’s list comprehensions and generators allow you to achieve many of the same tasks that you would use LINQ for in C#. In this post, we’ll look at Python equivalents for some of the most popular LINQ extension methods. We’ll mostly be looking at Python’s built-in functions and itertools module.

For these examples, our test data will be a list of fruit. But all of these techniques work with any interable, including the output of generator functions. Here’s our Python test data

fruit = ['apple', 'orange', 'banana', 'pear', 
         'raspberry', 'peach', 'plum']

Which of course in C# is

var fruit = new List<string>() { "apple", "orange",
 "banana", "pear", "raspberry", "peach", "plum" };

Any & All

LINQ’s Any method allows you to test whether any of the items in a sequence fulfil a certain requirement, while All checks if all of them do. Python’s built-in functions are named the same, so it’s really straightforward. Let’s see if any of our fruit contain the letter “e”, then see if all of them do:

>>> any("e" in f for f in fruit)
>>> all("e" in f for f in fruit)

in LINQ:

fruit.Any(f => f.Contains("e"));
fruit.All(f => f.Contains("e"));

Min & Max

Again, Python has built-in functions similarly named to LINQ. Let’s find the minimum and maximum fruit lengths:

>>> max(len(f) for f in fruit)
>>> min(len(f) for f in fruit)

which are the equivalents of:

fruit.Max(f => f.Length);
fruit.Min(f => f.Length);

Take, Skip, TakeWhile & SkipWhile

LINQ’s Take and Skip methods are very useful for paging data, or limiting the amount you process, and TakeWhile and SkipWhile come in handy from time to time as well (TakeWhile can be a good way of checking for user cancellation).

Take and Skip can be implemented using the itertools islice function. We can specify an end index, or a start and end index. If the end index is None, that means keep going to the end of the iterable. I’d prefer methods actually called “skip” and “take” as I think that makes for more readable code, but they could be easily created if needed.

Here’s Take(2) and Skip(2) implemented with Python. Since islice returns a generator function, I turn it into a list for debugging purposes:

>>> from itertools import islice
>>> list(islice(fruit, 2))
['apple', 'orange']
>>> list(islice(fruit, 2, None))
['banana', 'pear', 'raspberry', 'peach', 'plum']

islice does have the benefit though of letting you combine a skip and a take into one step rather than chaining them like you would in C#:


with islice:

>>> list(islice(fruit, 2, 4))
['banana', 'pear']

The itertools module does include a “takewhile” method and for LINQ’s SkipWhile, it’s “dropwhile”. With these functions, you might want to use Python’s lambda syntax, which is a rare example of where the Python is less succinct than C#.

>>> from itertools import takewhile
>>> list(takewhile(lambda c: len(c) < 7, fruit))
['apple', 'orange', 'banana', 'pear']
>>> from itertools import dropwhile
>>> list(dropwhile(lambda c: len(c) < 7, fruit))
['raspberry', 'peach', 'plum']

Here’s the same TakeWhile and SkipWhile in C#:

fruit.TakeWhile (f => f.Length < 7);
fruit.SkipWhile (f => f.Length < 7);

First, FirstOrDefault, & Last

With LINQ you can easily get the first item from an IEnumerable. This throws an exception if the sequence is empty, so FirstOrDefault can be used alternatively. With Python, the “next” method can be used on an iterable (but not on a list). Let’s use Python to get the first fruit starting with “p” and to return a default value when our generator looking for the first fruit starting with “q” doesn’t find any elements.

>>> next(f for f in fruit if f.startswith("p"))
>>> next((f for f in fruit if f.startswith("q")), "none")

There does not seem to be any built-in Python function to implement LINQ’s “Last” or “LastOrDefault” methods, but you could quite easily create one. Here’s a fairly rudimentary one:

>>> def lastOrDefault(sequence, default=None):
...     lastItem = default
...     for s in sequence:
...         lastItem = s
...     return lastItem
>>> lastOrDefault((f for f in fruit if f.endswith("e")))
>>> lastOrDefault((f for f in fruit if f.startswith("x")), "no fruit found")
'no fruit found'
You could do the same if you really needed the LINQ “Single” or “SingleOrDefault” methods, which also have no direct equivalent.


The LINQ Count extension method lets you count how many items are in a sequence. For example, how many fruit begin with ”p”?

fruit.Count(f => f.StartsWith("p"))
Probably the most logical expectation would be that Python’s “len” function would do the same, but you can’t call len on an iterable. There is a neat trick though you can use with the “sum” built-in function.
>>> sum(1 for f in fruit if f.startswith("p"))

Select & Where

We saw in the last blog post that a list comprehension already includes the capabilities of LINQ’s Select and Where, but there may be times you want to them to be available as functions. Python’s “map” and “filter” function take an iterable and a lamba and return an iterator (this is Python 3 only – in Python 2 they returned lists). Here’s a couple of simple examples of them in action, with the output turned into a list for debug purposes:

>>> list(map(lambda x: x.upper(), fruit))
>>> list(filter(lambda x: "n" in x, fruit))
['orange', 'banana']



At first glance it might appear that itertools groupby method behaves the same as LINQ’s GroupBy, but there is a gotcha. Python’s groupby expects the incoming data to be sorted by the key, so you have to call sorted first. This example shows us first trying to group without sorting (resulting in two “p” groups), and then doing it the right way. We’re grouping by first letter of the fruit, and I’m using a helper method to print out the contents of the grouped data:

>>> def printGroupedData(groupedData):
...     for k, v in groupedData:
...         print("Group {} {}".format(k, list(v)))
>>> from itertools import groupby
>>> keyFunc = lambda f: f[0]
>>> printGroupedData(groupby(fruit, keyFunc))
Group a ['apple']
Group o ['orange']
Group b ['banana']
Group p ['pear']
Group r ['raspberry']
Group p ['peach', 'plum']
>>> sortedFruit = sorted(fruit, key=keyFunc)
>>> printGroupedData(groupby(sortedFruit, keyFunc))
Group a ['apple']
Group b ['banana']
Group o ['orange']
Group p ['pear', 'peach', 'plum']
Group r ['raspberry']


As we saw above, the “sorted” built-in function in Python can be used to order a sequence. It returns a list, but this is understandable since to implement OrderBy it must iterate through the entire sequence first. Here we sort the fruit by their string length:

>>> sorted(fruit, key=lambda x:len(x))
['pear', 'plum', 'apple', 'peach', 'orange', 'banana', 'raspberry']


As far as I can tell there isn’t a built-in function in Python to emit a distinct iterable sequence, but the easiest way is probably to just construct a set. If you wanted to create a generator function, allowing you to abort early before reaching the end of a sequence, you could create your own helper method:

def distinct(sequence):
    seen = set()
    for s in sequence:
        if not s in seen:
            yield s


The last example I’ll look at is the Zip method. In Python there is an equivalent zip function, and it is actually a little simpler as it assumes you want a tuple, rather than LINQ’s where you need to explicitly create a result selector function. It actually supports zipping more than two sequences together which is nice. As with LINQ’s Zip, the resulting sequence is the length of the shortest. Here’s a quick example of the Python zip function in action:

>>> recipes = ['pie','juice','milkshake']
>>> list(zip(fruit,recipes))
[('apple', 'pie'), ('orange', 'juice'), ('banana', 'milkshake')]
>>> list(f + " " + r for f,r in zip(fruit,recipes))
['apple pie', 'orange juice', 'banana milkshake']


As can be seen, most of the main LINQ extension methods have fairly close Python equivalents, and those that don’t could be quite easily recreated. I don’t pretend to be an expert on Python, so if I’ve missed any cool tricks, let me know in the comments.

Thursday, 6 March 2014

Python List Comprehensions and Generators for C# Developers

If you’re a C# programmer and you’ve used LINQ, you’ll know how powerful it is to allow you to manipulate sequences of data in all kinds of interesting ways, without needing to write for loops. Python has similar capabilities, using what are called “list comprehensions” and “generators”. In this post, I’ll demonstrate how they work, showing them side by side with roughly equivalent C# code.

List Comprehensions

A list comprehension in Python allows you to create a new list from an existing list (or as we shall see later, from any “iterable”).

Let’s start with a simple example at the Python REPL. Here we create a list, that contains the square of each number returned by the range function (which in this case returns 0,1,2,…9)

>>> [x*x for x in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

This is equivalent to a C# LINQ statement that takes a range (using Enumerable.Range), selects the square (using Select), and then turns the whole thing into a list (using ToList):

Enumerable.Range(0, 10).Select(x => x*x).ToList();

Python list comprehensions also allow you to filter as you go, by inserting an “if” clause. Here, we’ll only take the squares of odd numbers:

>>> [x*x for x in range(10) if x%2]
[1, 9, 25, 49, 81]

This is equivalent to chaining a Where clause into our LINQ statement:

Enumerable.Range(0, 10).Where(x => x%2 != 0)
    .Select(x => x*x).ToList();

You can actually have two “for” clauses inside your list comprehension, so you could create some coordinates as a tuple like this:

>>> coords = [(x,y) for x in range(4) for y in range(4)]
[(0, 0), (0, 1), (0, 2), (0, 3), 
 (1, 0), (1, 1), (1, 2), (1, 3), 
 (2, 0), (2, 1), (2, 2), (2, 3), 
 (3, 0), (3, 1), (3, 2), (3, 3)]

The same effect can be achieved using the SelectMany clause in LINQ:

Enumerable.Range(0,4).SelectMany(x => Enumerable.Range(0,4)
    .Select(y => new Tuple<int,int>(x,y))).ToList();

You can see that the LINQ gets a little cumbersome at this point, although you can use the alternative syntax:

from x in Enumerable.Range(0,4)
from y in Enumerable.Range(0,4)
select new Tuple<int,int>(x,y)

Here's another Python list comprehension with two for expressions, making a list of all the spaces on a chessboard

>>> [x + str(y+1) for x in "ABCDEFGH" for y in range(8)]
['A1', 'A2', 'A3', 'A4', 'A5', 'A6', 'A7', 'A8', 
 'B1', 'B2', 'B3', 'B4', 'B5', 'B6', 'B7', 'B8',
 'C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 
 'D1', 'D2', 'D3', 'D4', 'D5', 'D6', 'D7', 'D8', 
 'E1', 'E2', 'E3', 'E4', 'E5', 'E6', 'E7', 'E8', 
 'F1', 'F2', 'F3', 'F4', 'F5', 'F6', 'F7', 'F8', 
 'G1', 'G2', 'G3', 'G4', 'G5', 'G6', 'G7', 'G8', 
 'H1', 'H2', 'H3', 'H4', 'H5', 'H6', 'H7', 'H8']

And in C#, you'd do something like:

"ABCDEFGH".SelectMany(x => Enumerable.Range(1,8)
    .Select(y => x+y.ToString())).ToList()

Dictionaries and Sets

You don't actually have to create lists. Python lets you use a similar syntax to create a set (no duplicate elements), or a dictionary. Here we'll start with a list of fruit, then use a list comprehension to make a list of string lengths. Then we'll make a set of unique fruit lengths, and we'll finally make a dictionary keyed on fruit name, and containing the length as a value:

>>> fruit = [‘apples’,’oranges’,’bananas’,’pears’]
>>> [len(f) for f in fruit]
[6, 7, 7, 5]
>>> {len(f) for f in fruit}
set([5, 6, 7])
>>> {f:len(f) for f in fruit}
{‘bananas’:7, ‘oranges’:7, ‘pears’:5, ‘apples’:6} 

We can create the set of unique lengths in C# by creating a HashSet, passing in our LINQ statement to its constructor. And you can use LINQ's ToDictionary extension method to make the equivalent dictionary of strings to lengths:

var fruit = new [] { "apples", "oranges", "bananas", "pears" };
fruit.Select(f => f.Length).ToList();
new HashSet<int>(fruit.Select(f => f.Length));
fruit.ToDictionary(f => f, f => f.Length);


Python generators are essentially the same concept as a C# method that returns an IEnumerable<T>. In fact, the syntax for creating them is very similar – you just need to use the yield keyword. Here’s a generator function that returns the names of my children:

def generateChildren():
    yield "Ben"
    yield "Lily"
    yield "Joel"
    yield "Sam"
    yield "Annie"

And here’s the same thing in C#:

public IEnumerable<string> GenerateChildren() 
    yield return "Ben";
    yield return "Lily";
    yield return "Joel";
    yield return "Sam";
    yield return "Annie";

Like with C#, Python generators uses lazy evaluation. This means that they could return infinite sequences. And it also means that it is not until we actually evaluate them that we will get any errors. This code example:

def generateNumbers():
    yield 2/2
    yield 3/1
    yield 4/0 # will cause a ZeroDivisionError
    yield 5/-1

numbersGenerator = generateNumbers()
print("Numbers Generator", numbersGenerator)
    numbers = [n for n in numbersGenerator]
    print("Numbers", numbers)
except ZeroDivisionError:

Generates the following output:

Numbers Generator <generator object 
    generateNumbers at 0x0000000002ADD4C8>

Python provides a method called “next” that allows you to step through the outputs from a generator one by one. Let’s try that with our children generator function:

>>> children = generateChildren()
>>> next(children)
>>> next(children)
>>> next(children)
>>> next(children)
>>> next(children)
>>> next(children)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

You’ll notice that calling next after we have reached the end gives us a StopIteration exception. C#’s closest equivalent to the Python next function is getting the enumerator and stepping through with MoveNext:

var children = GenerateChildren().GetEnumerator();

This produces the following output (the last item is repeated because we didn’t check the return code of MoveNext which indicates whether we reached the end of the enumeration).


In practice in C# it is fairly rare to use the enumerator directly. When you have an IEnumerable<T> you typically use it in a foreach loop or with some of the LINQ extension methods.

The Python list comprehension syntax also allows us to create new generators from existing generators. For example:

>>> (x*x for x in range(10))
<generator object <genexpr> at 0x0000000002ADD750>

This allows you to compose complex generators out of simple statements, creating a pipeline very much like you can with chained LINQ extension methods.


As you can see, Python list comprehensions and generators provide the same power that you are used to with C# and LINQ, and with a syntax that is more compact in most cases. Look out for a follow-up post shortly where I will demonstrate how many of the standard LINQ extension methods such as Any, All, Max, Min, Take, Skip, TakeWhile, GroupBy, First, FirstOrDefault, and OrderBy can be achieved in Python.