Visual Studio Team System Test Edition provides extremely powerful framework for developing load tests for web applications. The framework (Load Testing Framework) introduces a simple process for generating synthetic load, possibly from multiple machines, and provides rich infrastructure for generating reports.
The framework can be used to generate load and collect results for different purposes and for different kind of applications. It can be used to test web services in typical load (load testing) or extreme stress (stress testing), to measure performance and track degradation between releases (performance testing), to ensure that the service scales well when adding more resources (scalability testing) and to calculate the required resources for a given user workload (capacity testing).
In this post we will create a Load Test for the book store service that was introduced in the post Developing Silverlight 4.0 Three Tiers App with MVVM and WCF Data Services (OData via ATOM).
Here's a quick video that shows the load test running and the auto-generated summary report.
The source code can be downloaded from http://bookstoreservice.codeplex.com/SourceControl/list/changesets
In the next posts we will see how the Load Testing Framework can be integrated with 'Cruise Control .NET' (CCNet) and how we can extend CCNet to detect performance degradation between change-sets.
The System Under Test
Here's a high level view of the book store service
The service exposes OData (HTTP) interface through which its clients can query/update books. Persistence is achieved through a repository abstraction, as it is now, for the sake of simplicity we'll use a simple file based repository.
Here's the classes diagram:
Typically, WCF data Services expose data through Entity Framework, in such cases the DataService class is implemented by Entity Framework's auto generated class that exposes the appropriate metadata classes and takes care of all CRUD operations.
In our case, we defined custom metadata classes (Book, Author) and used Reflection Provider for CRUD. To make this work, we inherited the DataService class via BookStoreDataServiceCore, and implemented the IDataServiceUpdateProvider interface via BookStoreUpdateProvider. The class FilePersistence is the one that actually stores the data into a file.
Developing Functional Tests
To simplify the setup process - we will host the data service in the test process. To make this work, we'll use the TestService class to host and launch the service.
public class TestService : IDisposable
{ private WebServiceHost host;private readonly Uri serviceUri;
private static int s_lastHostId = 1;
private const int NumberOfAttempts = 10;
public TestService(Type serviceType) { int attempts = 0; for (;;) {int hostId = Interlocked.Increment(ref s_lastHostId);
string hostName = "BookStoreDataService" + hostId;
serviceUri = new Uri( "http://localhost/Temporary_Listen_Addresses/" + hostName); host = new WebServiceHost(serviceType, serviceUri); try {host.Open();
break;}
catch (Exception e) {host.Abort();
host = null; if (attempts++ >= NumberOfAttempts) {string message = string.Format(
"Could not open a service after {0} attempts", NumberOfAttempts);
throw new InvalidOperationException(message, e);
}
}
}
}
public void Dispose()
{if (host != null)
{host.Close();
host = null;}
}
public Uri ServiceUri { get { return serviceUri; }}
}
End users of the book store service use it to query for books, query for authors and update books metadata. As such, we have 3 tests, one for each user scenario.
[TestClass]
public class FunctionalTests
{private static TestService s_service;
private DataServiceContext serviceContext; // Run once for all the tests[ClassInitialize()]
public static void ClassInitialize(TestContext testContext)
{ // Start the DataService (Using WebServiceHost) s_service = new TestService(typeof(BookStoreDataService));
}
[ClassCleanup]
public static void ClassCleanup()
{if (s_service != null)
{s_service.Dispose();
s_service = null;}
}
// Run before every test[TestInitialize]
public void TestInitialize()
{ // Create the client data context serviceContext = new DataServiceContext(s_service.ServiceUri);}
[TestMethod]
public void QueryForBooks()
{ testContextInstance.BeginTimer("Read Books"); DataServiceQuery<Book> booksQuery = serviceContext.CreateQuery<Book>("Books");List<Book> Books = booksQuery.ToList();
testContextInstance.EndTimer("Read Books"); Assert.AreEqual(2, Books.Count, "Books number is wrong");Book book = Books[0];
Assert.AreEqual("Book1", book.Name, "Book name is wrong");
book = Books[1];
Assert.AreEqual("Book2", book.Name, "Book name is wrong");
}
[TestMethod]
public void QueryForAuthors()
{ testContextInstance.BeginTimer("Read Authors"); List<Author> tables = serviceContext.CreateQuery<Author>("Authors").ToList(); Assert.AreEqual(3, tables.Count, "Authors number is wrong"); testContextInstance.EndTimer("Read Authors");}
[TestMethod]
public void UpdateBook()
{ List<Book> Books = serviceContext.CreateQuery<Book>("Books").ToList(); Assert.AreEqual(2, Books.Count, "Books number is wrong");Book book = Books[0];
Assert.AreEqual("Book1", book.Name, "Book name is wrong");
const string description = "New description";
book.Description = description;
testContextInstance.BeginTimer("Update Book");serviceContext.UpdateObject(book);
// Synchronous saveserviceContext.SaveChanges(SaveChangesOptions.None);
testContextInstance.EndTimer("Update Book"); serviceContext = new DataServiceContext(s_service.ServiceUri); Books = serviceContext.CreateQuery<Book>("Books").ToList();book = Books[0];
Assert.AreEqual(book.Description, description, "Update description failed");}
#region TestContext private TestContext testContextInstance; /// <summary> ///Gets or sets the test context which provides ///information about and functionality for the current test run. ///</summary> public TestContext TestContext {get
{ return testContextInstance;}
set
{ testContextInstance = value;}
}
#endregion}
As you can see, we are also wrapping interesting transactions with TestContext.BeginTimer(scenario), TestContext.EndTimer(scenario) – this will be useful when these test will run from the context of a load test.
Creating a Load Test
Creating a load test is actually pretty simple once we have the functional tests in place.
First add new 'Load Test' to your test project.
In the wizard, set the number of concurrent users.
Add the the functional tests to the 'Test Mix'
Add performance counters. Here we will probably want to add the required counters on the machine/s that run the system under test,
Click finish. A new loadtest file will be added to the project.
Now, we need to define appropriate frequency for each test. Double Click on the loadtest file, right click on 'Test Mix' and select 'Edit Test Mix'
We set the frequency for QueryForAuthors to 25%, QueryForBooks to 70% and UpdateBook to 5%. When testing a live production service - we would probably want to get some telemetry from production to understand the real workload.
Now, we can go ahead a run the load test. By default, the load test will run locally and the results will be saved to SQL Express.
Generating Load from Multiple Agents
For some web services, generating load from one machine is sufficient. In such cases, we can use the default configuration and have tests running without a Controller.
In cases where we need to generate load from multiple machines, we can setup a Test Controller and multiple Test Agents such that the tests will run in a distributed fashion from multiple machines.
This link will guide you though the process of installing and setting up the Controller/Test Agents and updating the TestSettings such that the tests will use the Controller instead of the default local execution engine.
Storing the Results
By default, when test run locally, the results are stored into SQL Express Database. The Database 'LoadTest2010' that's used to store the results is created in the first run
To connect to this database using SQL Server Management Studio, set .\SQLExpress in the 'Server name' and click connect.
To use SQL Server in local runs, we need change the Connection String setting in Visual Studio and setup the database ourselves, This link will guide you through the entire process of setting up the database and changing the Connection String setting in Visual Studio.
nice!
ReplyDeletekeep up the good work
y.t ashahim