Monty’s Gush

How to make several asynchronous requests in parallel to the same web service, when the number of such requests is dynamically determined.

Thanks to AsyncControllers, it’s quite easy to make several asynchronous requests in parallel to different web services, and have MVC gather the results.

A well-written example is on MSDN. Briefly, you save the result of each async call into the Parameters collection of the AsyncManager like so:

AsyncManager.Parameters["headlines"] = e.Value;
...
AsyncManager.Parameters["scores"] = e.Value;
...
AsyncManager.Parameters["forecast"] = e.Value;

Your Completed method will then bind to the parameters you have named:

public ActionResult IndexCompleted(
    HeadlinesResult headlines,
    ScoresResult scores,
    ForecastResult forecast) { ... }

This is fine if you know how many requests you want to make. But what if you want to make several requests in parallel to the same web service (for example, with slightly different parameters) — but you don’t know in advance how many requests you will need to make?

For example, you might show the weather forecast for as many days into the future as the user requests, up to seven days, with each day requiring a separate request to the forecast service.

In this case you need a dynamically-sized collection of results of the same type, such as a list. Ideally, we want our Completed method to look something like this:

public ActionResult IndexCompleted(List results) { ... }

Fist of fun

However, we don’t know on which threads, or even in what order, this collection will be added to when each of the web service calls complete independently.

The System.Collections.Concurrent namespace gives us a more suitable type to withstand the mutation this collection will have to put up with.

public class ForecastController : AsyncController
{
    public void IndexAsync(ForecastInputModel input)
    {
        var results = new ConcurrentBag<ForecastResult>();

        AsyncManager.Parameters["results"] = results;

        foreach (var request in GetForecastRequests(input))
        {
            AsyncManager.OutstandingOperations.Increment();

            var s = new WeatherService();

            s.GetForecastCompleted += (sender, e) =>
                {
                    results.Add(e.Value); // threadsafe
                    AsyncManager.OutstandingOperations.Decrement();
                };
            
            s.GetForecastAsync(request);
        }
    }

    public ActionResult IndexCompleted(ConcurrentBag<ForecastResult> results)
    {
        ...
    }  
 }

Enjoy what you saw.

I don’t use the famous Albahari PredicateBuilder because it doesn’t work so well with Entity Framework or the latest version of NHibernate.

This is because it relies on InvocationExpressions, which aren’t supported by many query translators, and the workaround requires an unnatural call to Tomas‘ AsExpandable method in all your queries.

A couple of years ago, Colin Meek on the EF team showed me how to compose predicates without using InvocationExpressions and since then I’ve used my own synthesized version of PredicateBuilder, with the same API as the awesome Albahari original.

If you need a PredicateBuilder that works as you’d expect with Entity Framework, NHibernate… as well as Linq to SQL, here it is. (It ought to work in Silverlight too. Let me know…!)

    /// <summary>
    /// Enables the efficient, dynamic composition of query predicates.
    /// </summary>
    public static class PredicateBuilder
    {
        /// <summary>
        /// Creates a predicate that evaluates to true.
        /// </summary>
        public static Expression<Func<T, bool>> True<T>() { return param => true; }

        /// <summary>
        /// Creates a predicate that evaluates to false.
        /// </summary>
        public static Expression<Func<T, bool>> False<T>() { return param => false; }

        /// <summary>
        /// Creates a predicate expression from the specified lambda expression.
        /// </summary>
        public static Expression<Func<T, bool>> Create<T>(Expression<Func<T, bool>> predicate) { return predicate; }

        /// <summary>
        /// Combines the first predicate with the second using the logical "and".
        /// </summary>
        public static Expression<Func<T, bool>> And<T>(this Expression<Func<T, bool>> first, Expression<Func<T, bool>> second)
        {
            return first.Compose(second, Expression.AndAlso);
        }

        /// <summary>
        /// Combines the first predicate with the second using the logical "or".
        /// </summary>
        public static Expression<Func<T, bool>> Or<T>(this Expression<Func<T, bool>> first, Expression<Func<T, bool>> second)
        {
            return first.Compose(second, Expression.OrElse);
        }

        /// <summary>
        /// Negates the predicate.
        /// </summary>
        public static Expression<Func<T, bool>> Not<T>(this Expression<Func<T, bool>> expression)
        {
            var negated = Expression.Not(expression.Body);
            return Expression.Lambda<Func<T, bool>>(negated, expression.Parameters);
        }

        /// <summary>
        /// Combines the first expression with the second using the specified merge function.
        /// </summary>
        static Expression<T> Compose<T>(this Expression<T> first, Expression<T> second, Func<Expression, Expression, Expression> merge)
        {
            // zip parameters (map from parameters of second to parameters of first)
            var map = first.Parameters
                .Select((f, i) => new { f, s = second.Parameters[i] })
                .ToDictionary(p => p.s, p => p.f);

            // replace parameters in the second lambda expression with the parameters in the first
            var secondBody = ParameterRebinder.ReplaceParameters(map, second.Body);

            // create a merged lambda expression with parameters from the first expression
            return Expression.Lambda<T>(merge(first.Body, secondBody), first.Parameters);
        }

        class ParameterRebinder : ExpressionVisitor
        {
            readonly Dictionary<ParameterExpression, ParameterExpression> map;

            ParameterRebinder(Dictionary<ParameterExpression, ParameterExpression> map)
            {
                this.map = map ?? new Dictionary<ParameterExpression, ParameterExpression>();
            }

            public static Expression ReplaceParameters(Dictionary<ParameterExpression, ParameterExpression> map, Expression exp)
            {
                return new ParameterRebinder(map).Visit(exp);
            }

            protected override Expression VisitParameter(ParameterExpression p)
            {
                ParameterExpression replacement;

                if (map.TryGetValue(p, out replacement))
                {
                    p = replacement;
                }

                return base.VisitParameter(p);
            }
        }
    }

A commenter recently discovered that the technique I use to generate a cache key for any IQueryable was not working properly for one of his queries. The query contained a call to List.Contains, and the contents of the list was not getting put into the cache key.

The problem was that the query cache didn’t specifically know about local collection values.

Local collection values can be supplied as parameters to query operators such as Contains and Any, if the query provider supports them.

var ids = new List<int> { 1, 3, 5 };

var q = from c in db.Customers
        where ids.Contains(c.CustomerId)
        select c;

As far as FromCache was concerned, ids is just a constant expression, and treated it no differently to any other constant when evaluating its cache key.

Unfortunately, the ToString implementation of Lists (and of Arrays, and IEnumerables in general) is not suitable for use as a cache key, because it outputs the type name and not (for example) a representation of every element in the collection.

I couldn’t understand how I’d overlooked this. But then I remembered that Entity Framework v1, which was the main provider I was using at the time, didn’t support local collections either!

I’d actually written my own WhereAny and WhereAll methods, which build up expressions from local collections and produce queries that Entity Framework v1 could understand. But that post never made it out of my WordPress drafts.

LINQ to SQL has always supported the Contains method, and Entity Framework now supports both Any and Contains overloads with local collections. So, I’ve updated the original source code and the blog post to support local collections.

This involves a pass over the expression tree to expand appropriate local collection values which might be used in methods like Any or Contains. Each method call in the expression tree is examined, and the argument to any parameters of type IEnumerable<> or List<> is wrapped in an object with an implementation of ToString which prints every element in the collection.

I’ve also extracted the method IQueryable.GetCacheKey and made it public, which could help emphasise that the meat of the idea is to automatically generate a cache key for a query (and that you can do whatever you like with it, such as make a FromCache extension method, or managing the cache manually).

If you’re using the FromCache extension method, please update use the latest version of the source code!

The generic IEnumerable interface is the most important abstraction in the .NET framework after System.Object.

If you’re not 100% comfortable with IEnumerable and IEnumerable<T>, this introductory article might help!

Prerequisite – what’s an interface?

An interface, in the general sense, is just the publicly exposed signature of a type, where ‘type’ means class or struct in C#.

We can describe an interface by writing out the full public signature of a type.

public class Customer

{

public int Age{ get; }

public void MakeHappy();

// … more methods and properties!

}

This is not valid C#, but all types have “an interface” in this basic sense. Right click on string in your code and click Go to definition to see what I mean.

.NET interface types

Java and .NET also support a programming construct called an interface type, defined using the interface keyword. This is a formalization of the general OO concept of “interface”.

What? An interface type is really just an abstract class guaranteed to contain no implementation code.

Why? To allow a limited form of multiple inheritance.

When you “inherit” from an interface, you declare that you support the interface’s methods. This means your type can be used in situations that only require that it supports a given interface. I practice, this functionality is not directly related to the main job of the type. Interfaces give you a way of mixing and composing functionality into a type, allowing it to be used in more situations.

Common .NET interfaces

IComparable – an object that can be compared with other objects of the same type

IConvertible - an object that can be converted to different types

ISerializable - an object that can be serialized

IDisposable - an object that can deterministically clean up after itself

By convention, .NET interface type names are prefixed with I. Unfortunately this makes them less friendly and intuitive than normal classes. Just pretend the ‘I’ isn’t there when you read them, and keep in mind that interface types are simply a special kind of abstract class.

So, interface types can just be used in the same way as other types. In particular, you can declare variables and parameters of interface types.

bool IsABiggerThanB(IComparable a, IComparable b)

{

return a.CompareTo(b) > 0;

}

Here, parameters a and b can be of any type* that implements IComparable, which means they must define a CompareTo method.

[* Footnote. In this case, a and b have have to be the same type as each other, or a runtime exception will be thrown by CompareTo.]

The CompareTo method returns 1 if the object is bigger, 0 if the same, and -1 if smaller than the object passed to it. Many types in the .NET Framework support IComparable. For example, you can compare strings to sort them into alphabetical order.

Notice how inflexible this would be without interface types. We’d either have to

(1) inherit from some kind of “Comparable” class, using up our only base class, or

(2) support comparability in an ad-hoc, incompatible way for each type (making the polymorphic code above impossible).

Where are we?

Interface types support rich object oriented polymorphism without the burden of full-blown multiple inheritance.

The Iterator design pattern

Provides a way to traverse the elements of an aggregate object without exposing the underlying representation.

[GoF, 1995]

Aggregate object = a data structure; a collection, list or sequence of things of the same type.

Why?

• You might need different kinds of traversal (e.g. forwards, backwards, in-order, pre-order).
• You might want to keep track of several traversals on the same collection at once.
• All collections, lists, sequences etc. should be traversable in the same way.

Sure, often a list can be traversed with a basic for loop. For example, we can keep track of the current position in a list with a local integer variable and index into an array.

for (int i = 0; i < 10; i++)

{

object current = myCollection[i];

}

But this will only work if the collection is the kind of collection that can be indexed in to, such as a simple list or array. It doesn’t make any sense to “index” into a binary tree, for example; nor into many other data structures that we might want to traverse.

A binary tree from math.bas.bg

What about infinite lists? What if the size were unknown?

Why can’t the collection manage its own traversal?

Depending on the kind of data structure, we’d probably need some kind private pointer inside the object to keep track of the current element, and some additional methods on the class itself. But you can see that this is not going to provide a solution to any of the above three requirements. This solution would allow one traversal at a time, and you would need to add additional methods to the class for each traversal algorithm you wanted to provide.

The Iterator design pattern in .NET

The Iterator pattern takes the responsibility for traversal out of the collection itself, and puts it into another object. This specialised object is responsible only for traversing the collection (in a particular way), and nothing else.

For example, an instance of this specialised object might contain a pointer to keep track of the current item, and some code implementing the particular traversal logic. The class will probably have intimate knowledge of the collection class that it supports traversal over, and might well be defined as a nested class.

The Iterator pattern is supported in the .NET framework v1 via two interface types in the System.Collections namespace.

IEnumerator – a specialised object responsible for doing the traversal

IEnumerable - a data structure that can be traversed

Microsoft uses ‘enumerate’ instead of ‘iterate’. So you can think instead of “the Enumerator pattern” if you like. Also note that this sense of ‘enumerate’ has nothing really to do with enumerated types (enums in C#).

public interface IEnumerator

{

object Current { get; }

bool    MoveNext();

}

This interface is the really the key to the pattern. It defines pretty much the simplest possible interface for traversing over a data structure. An enumerator object can tell you what the current item is, and you can tell it to move to the next item, and that’s it.(*) If MoveNext is false, we have run out of items. These two methods are just a slightly more verbose way of supporting the operation “Give me the next one“, which is the essence of the enumerator interface.

Notice the enumerator interface makes no assumptions at all about the structure, internal implementation or size of the data strucure that is being traversed. It’s perhaps the purest notion of a “sequence” you can think of.

(*) There are one or two other methods on the interface, but they’re not essential to understanding the pattern.

while (enumerator.MoveNext())

{

object current = enumerator.Current;

}

Note that Microsoft could have designed this type as an abstract class instead of an interface type.

Great – so now we can iterate over any kind of collection or sequence using this code, provided we have an enumerator for that kind of sequence.

If only we had a standardised way to get an enumerator!

public interface IEnumerable

{

IEnumerator GetEnumerator();

}

This is the public face of the Iterator pattern in .NET. It simply provides a well-known way to get an instance of a default enumerator for a particular sequence or collection.

So, if I give you an object that is enumerable, you can call GetEnumerator to give you a nice enumerator object that you can use to do a standard traversal of the object’s items.

The IEnumerable interface is implemented by many different types, including all collection types, and even the String class.

string s = “abcdef”;

IEnumerator enumerator = s.GetEnumerator();

while (enumerator.MoveNext())

{

Console.Write( ((char) enumerator.Current) + “.” );

}

// output: “a.b.c.d.e.f.”

Language-level support

It turns out that this usage pattern (obtaining the default enumerator, then calling MoveNext on it in a loop) is so common it is formalised into a helpful language-level construct in C# and VB.

If you’ve ever wondered how the foreach keyword works, now you know. The following code is identical to the previous example.

string s = “abcdef”;

foreach (char c in s)

{

Console.Write(c + “.”);

}

The C# compiler simply expands this code to that in the previous example. The intermediate language produced is identical.(*)

(*) OK, it’s not, but assume it is. Check out the difference in Reflector.

Where are we?

The Iterator design pattern is deeply supported in the .NET framework class library and languages.

Where next?

Generics in .NET 2.0 improved support for the pattern by adding strongly-typed versions of the interfaces, IEnumerator<T> and IEnumerable<T>. IEnumerable<T> in particular plays a key role in the language integrated query (LINQ) features and C# 3.0 in general. The C# 2.0 compiler was given the power to write its own iterator implementations via the yield keyword.

The XML Document Transform language, or XDT, shipped with Visual Studio 2010. It’s an XML dialect designed to help you automatically transform your .NET config files appropriately for your various deployment environments.

Great. I love XML, obviously

Don’t worry, XDT is very simple and specifically designed for the task of transforming configuration files. For this scenario, it is much easier to use than a general-purpose transformation language like XSLT.

Application configuration files tend to be mostly identical across different deployment environments. Only small modifications, such as the value of your connection strings, the debug attribute, and so on, are required to create a production Web.config file out of a development Web.config file.

To show how simple it is, the basic semantics of the language can be expressed in a few lines of C#.

Let’s assume an existing Web.config file for our input document. We’ll also write a small XDT transform document which sets the ASP.NET debug attribute to false:

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
	<system.web>
		<compilation debug="false" xdt:Transform="SetAttributes" />
	</system.web>
</configuration>

Notice it looks a bit like a small Web.config document. The input doc could be a big, complex Web.config file, but that doesn’t matter – the transform doc only needs to contain the differences.

The Transform attribute is defined within a distinguished XML namespace to ensure that it can’t clash with any existing attributes in the input doc. In this case, we’re using one of the most useful built-in transforms, SetAttributes, which modifies attribute values on the targeted element to those that are specified in the transform doc – in this case, setting debug to false. There are other useful transforms, such as Remove and Replace, which alter the targeted element in different ways.

A transform implicitly specifies the element(s) in the input document to act upon by virtue of the element that it belongs to: expressed in XPath, the above transform would target the
/configuration/system.web/compilation element(s).

Here is a simple implementation of XDT using C#:

        public XDocument Transform(XDocument inputDoc, XDocument transformDoc)
        {
            var workingDoc = new XDocument(inputDoc);

            // (1) pair each "Transform" element with the element(s)
            // it targets in the working document
            var xs = from e in transformDoc.Descendants()
                     from a in e.Attributes(Namespaces.Xdt + "Transform")
                     let xpath = GetElementXPath(e)
                     select new
                     {
                         TransformElement = e,
                         TargetElements = workingDoc.XPathSelectElements(xpath)
                     };

            // (2) apply each transform to its target elements
            foreach (var x in xs)
            {
                ApplyTransform(x.TransformElement, x.TargetElements);
            }

            return workingDoc;
        }

Show me the source code

This is a slight simplification, and of course doesn’t show you the implementation of the individual transforms or how to compute the XPath. You can find the full, reasonably complete implementation of XDT on Google Code.

Locators

The implicit XPath location of, for example, a /configuration/appSettings/add element would usually specify more than one element in most Web.config files. While this is valid, it’s not very useful. XDT provides a second attribute, Locator, which allows you to augment the implicit XPath expression with a predicate to filter the target element set:

                <configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
                  <appSettings>
                    <add xdt:Locator="Condition(@key='key3')" />
                  </appSettings>
                </configuration>

This would give us target XPath of /configuration/appSettings/add[@key='key3'], which is much more useful (although it doesn’t do anything, because we haven’t specified any transform!).

A convenience case of the Condition locator is to simply Match on the specified attribute, so to actually alter a particular app setting value, you would write:

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
	<appSettings>
		<add key="key3" value="new value" xdt:Locator="Match(key)" xdt:Transform="SetAttributes" />
	</appSettings>
</configuration>

This specifies a SetAttributes transform with the same target XPath as the previous example.

Why XDT?

• It’s part of Visual Studio 2010, and you can expect it to become the standard way of managing different config files.
• It’s really rather nice – it’s ideal for tweaking XML documents, so long as the changes aren’t big. It’s very readable (despite being XML) and quite easy for anyone to use.
• You don’t really want to use XSLT for this sort of thing.

Why write a free implementation?

I wanted to use XDT in production before Visual Studio 2010 was released. I also find that thinking about the implementation of something helps you understand its semantics. However, there are some other good reasons:

• XDT is currently restricted to web applications. Out of the box, it can’t be used with Windows or console apps, or even web “site” style projects. But this is no restriction of the language itself.
• XDT is currently restricted to Web.config files. Real applications tend to have more than one XML configuration file which needs to be transformed.
• You might not be using Visual Studio 2010.
• You might want to decouple the notions of build configuration and build environment. They’re not the same concept, but the Visual Studio 2010 designers chose to keep things simple. With several “release” environments, this conceptual simplicity comes at the cost of flexibility and creates duplication in the project and solution manifest files. A more sophisticated build system might prefer to keep just the standard two build configurations (Debug, Release) but define several deployment environments (Dev, Test, Uat, Live…), each with different config transformations.

If you’re creating a serious build system, these restrictions would otherwise probably prevent you from using XDT for your transformations, which would be a shame.

It feels like a classic Microsoft situation – a well-engineered core technology, but restricted (understandably) for commercial / shipping reasons. If you do want to go ahead and use this alternative implementation, it’s very straightforward to create an MSBuild task to call XdtTransformer.Transform() in order to use XDT in your build system. If there’s enough demand, I could add one to the distribution on Google Code.

Disclaimer The semantics of this implementation are based purely on my reading of the XDT product documentation on MSDN. The implementation passes all of the example tests I’ve found on the web – but, as usual, no warranties are implied.

A generic function to group a sequence into sequences of equal size.

var numbers = new [] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 };

var groups = numbers.GroupsOf(3);

Now groups is a sequence of sequences containing three items.

Perhaps you might use this in some tricky UI scenario or to gamble away your integers. You’ll need the IEnumerator.Next extension too.

Is this a special case of a more general function, I wonder?

/// <summary>
/// Groups a sequence into sequences of the specified size.
/// The final group may be shorter if there are leftover
/// elements.
/// </summary>
/// <param name="count">The size of the groups.</param>
public static IEnumerable<IEnumerable<T>> GroupsOf<T>(
    this IEnumerable<T> source,
    int count)
{
    using (var it = source.GetEnumerator())
    {
        while (true)
        {
            var group = it.Next(count).ToList();

            if (group.Any())
                yield return group;
            else
                break;
        }
    }
}

I still find it remarkable how syntax alone can alter the way I conceptualise a problem.

Here’s a useful helper for pulling the next n elements out of an enumerator. Think of it as a generalisation of MoveNext. I’ve found it handy for implementing certain IEnumerable extensions.

public static class EnumeratorExtensions
{
    /// <summary>
    /// Returns a specified number of contiguous elements
    /// from the enumerator.
    /// </summary>
    public static IEnumerable<T> Next<T>(
        this IEnumerator<T> source,
        int count)
    {
        int counter = 0;

        while (counter < count)
        {
            if (source.MoveNext())
                yield return source.Current;
            else
                break;

            counter++;
        }
    }
}