Web Notes

In today post I want to describe how to optimize usage of a ADO.NET SqlDataReader class by using multi-threading. To present that lets me introduce a problem that I will try to solve.  Scenario : In a project we decided to move all data from a multiple databases to one data warehouse. It will be a good few terabytes of data or even more. Data transfer will be done by using a custom importer program. Problem : After implementing a database agnostic logic of generating and executing a query I realized that I can retrieve data from source databases faster that I can upload them to big data store through HTTP client -importer program. In other words, data reader is capable of reading data faster then I can process it an upload to my big data lake. Solution : As a solution for solving this problem I would like to propose one of a multi-thread design pattern called Producer/Consumer . In general this pattern consists of a two main classes where: Producer class is respons

Today I want to cover a very common scenario about how to create a HTTP client for a WebAPI 2 service. To present my implementation I will be using one of a Task<T> extension method that I described recently on my blog. Let`s start from defining an API service. In this example it udes a REST based WebApi service with the following implementation.   [ AllowAnonymous ]     public   class   CityController    {         ///   <summary>         ///  The city repository.         ///   </summary>         private   readonly   ICityRepository  cityRepository;         ///   <summary>         ///  Initializes a new instance of the  <see cref= "CityController"  />  class.         ///   </summary>         ///   <param name= "cityRepository" > The city repository. </param>         public  CityController( ICityRepository  cityRepository)        {             this .cityRepository = cityRepository;        }