Skip to main content

Posts

Using Newtonsoft serializer in CosmosDB client

Problem In some scenarios engineers might want to use a custom JSON serializer for documents stored in CosmosDB.  Solution In CosmosDBV3 .NET Core API, when creating an instance of  CosmosClient one of optional setting in  CosmosClientOptions is to specify an instance of a Serializer . This serializer must be JSON based and be of  CosmosSerializer type. This means that if a custom serializer is needed this should inherit from CosmosSerializer abstract class and override its two methods for serializing and deserializing of an object. The challenge is that both methods from  CosmosSerializer are stream based and therefore might be not as easy to implement as engineers used to assume - still not super complex.  For demonstration purpose as or my custom serializer I'm going to use Netwonsoft.JSON library. Firstly a new type is needed and this must inherit from  CosmosSerializer.  using  Microsoft.Azure.Cosmos; using  Newtonsoft.Json; using  System.IO; using  System.Text; ///   <
Recent posts

Persisting Enum in database with Entity Framework

Problem statement We all want to write clean code and follow best coding practices. This all engineers 'North Star' goal which in many cases can not be easily achievable because of many potential difficulties with converting our ideas/good practices into working solutions.  One of an example I recently came across was about using ASP.NET Core and Entity Framework 5 to store Enum values in a relational database (like Azure SQL). Why is this a problem you might ask... and my answer here is that you want to work with Enum types in your code but persist an integer in your databases. You can think about in that way. Why we use data types at all when everything could be just a string which is getting converted into a desirable type when needed. This 'all-string' approach is of course a huge anti-pattern and a bad practice for many reasons with few being: degraded performance, increased storage space, increased code duplication.  Pre-requirements 1. Status enum type definition

Runtime generated objects serialization

If you think in a generic way about all well implemented RESTful APIs you will find a pattern that can be easily described and stored in metadata. Most RESTful API is just a combination of the following elements: Resource location (URL) HTTP method Header information Input parameters (required and optional) Content type Output parameters Business logic description Today I would like to discuss an interesting problem that I came across recently. Imagine for a second that you need to implement a RESTful API client which uses a combination of metadata which describes API and a user input in order to make a HTTP calls.  In such a scenario you will quickly realize that for a subset of API calls you will need to develop a custom classes in order to have  them later serialized (to JSON or XML) in runtime so that you can send it via POST or PUT requests. This rises a question. Do I really need to implement N - 1 classes that represent all types that some API(s) expect(s) as a

Deep dive in unit testing

These days each product reaching market is labelled as top quality - no matter if it`s a toy or a car or an application. Everyone talk about quality, quality is everywhere and at the same time quality by nature is a tricky thing to define and measure. To give you an example, imagine a two new brand cars from a two different car manufacturers like for example BMW and Fiat. Dealers of both brands will tell you that their cars are top quality and in fact that is true! The problem starts when you try to understand what top quality means for both car manufacturers - what are their standards of quality. What Fiat can consider as top quality might be completely not acceptable for BMW. From client perspective what really important is to understand how to measure quality in a standards driven way. As an example let's compare a European car safety performance assessment rating for both brands (NCAP is rated from 1 to 5 stars where 5 stars is given for most safety cars). In this rating ca

Using Hortonworks Hive in .NET

A few months ago I decided to learn a big data. This sounds very complex and of course it is. All these strange names which actually tells nothing to person who is new in these area combined with different way of looking at data storage makes entire topic even more complex. However after reading N blogs and watching many, many tutorials today I finally had a chance to try to write some code. As in last week I managed to setup a Hortonworks distribution of Hadoop today I decided to connect to it from my .NET based application and this is what I will describe in this post. First things first I didn`t setup entire Hortonworks ecosystem from scratch - I`d love to but for now it`s far beyond my knowledge thus I decided to use a sandbox environment provided by Hortonworks. There are multiple different VMs available to download but in my case I`ve choose a Hyper-V. More about setting this environment up you can read here . Picture 1. Up and running sandbox environment. Now whe

Multithread processing of the SqlDataReader - Producer/Consumer design pattern

In today post I want to describe how to optimize usage of a ADO.NET SqlDataReader class by using multi-threading. To present that lets me introduce a problem that I will try to solve.  Scenario : In a project we decided to move all data from a multiple databases to one data warehouse. It will be a good few terabytes of data or even more. Data transfer will be done by using a custom importer program. Problem : After implementing a database agnostic logic of generating and executing a query I realized that I can retrieve data from source databases faster that I can upload them to big data store through HTTP client -importer program. In other words, data reader is capable of reading data faster then I can process it an upload to my big data lake. Solution : As a solution for solving this problem I would like to propose one of a multi-thread design pattern called Producer/Consumer . In general this pattern consists of a two main classes where: Producer class is respons