Apache Kafka Terminology

Event Streaming, Programming

Broker

A Kafka broker is a server that stores and serves data. Typically, a Kafka cluster consists of multiple brokers to handle large scale and to provide fault tolerance.

Topics

A topic is a logical grouping of messages. Messages are sent from a producer and read by a consumer.

Partition

A partition is a way to divide a topic’s messages. Each partition is an ordered sequence of messages. By splitting a topic into partitions, it allows multiple consumers to read messages simultaneously. Partions are a fundamental way to enable parallelism. Messages are distributed between topics in two ways

  • Key-based partitioning: If a key is provided, Kafka uses the key to determine the partition
  • Round-Robin: If there is no key, messages are distributed evenly between the partitions.

Producer

Service that writes messages to Kafka topics.

Consumer

Service that reads messages from a given Kafka topic, to ensure messages ordering the services is single threaded. Usually, a consumer is part of a consumer group to enable parallel processing of messages. A consumer can have an optional Client Id, it is used for identification, logging, and monitoring.

Consumer Group

A consumer group is a collection of consumers that work together to read messages from a Kafka topic. Consumer groups enable parallel processing by having each consumer read messages from one partition, hence, guaranteeing messages ordering.

Message

A message consists of

  • A key, which is optional, used for partitioning
  • Data payload
  • Timestamp, when the record was created

Offset

The offset is a unique identifier for each message within a partition, the offset represents the message positions in the partition. Consumers use offsets to keep track of their position with a topic’s partition, this ensures that messages are not reprocessed. Consumers can commit their offsets either automatically or manually.

Storing incremental data in a document

Programming

Scenario

A computer system (the system) is receiving datafrom IoT devices connected to vehicles. The devices send data to the server every 30 seconds. The payload sent to the server contains:

  • Speed
  • Position
  • Timestamp
  • Ignition on/off

The devices capture data points every second, so for one payload sent to the server we have 30 positions, 30 speed values, etc.

The payload sent to the server is 1 488 bytes.

To be able to compose a complete trip, the system stores the state of the devices which includes the incremental data sent to the server. When the payload contains ignition on, a new trip is created, when the payload contains ignition off, the trip is composed of the stored state.

The state of the device is stored in a document database.

The state document

{ 
  "Id": string, 
  "Data": byte[], 
  "State": ["Driving", "Parked"]
}

The system updates and stores the document for each payload sent to the server, using ReplaceOneAsync

await _collection.ReplaceOneAsync(s => s!.Id == id, state, new ReplaceOptions { IsUpsert = true });

One device is registered in the system, and the vehicle completes 20 trips per day, each trip being 15 minutes long. The total data sent for one trip is 30 packages, 1488 bytes * 30 packages.

Payload size sent to the system from the device

Size (bytes)
One trip30 packages * 1488 bytes = 44 640 bytes
One dayOne trip * 20 = 892 800 bytes
One weekOne day * 07 = 6 249 600 bytes
One monthOne day * 30 = 26 784 000 bytes (26 MB)

The problem

The company is invoiced 780 MB for data ingress from the cloud provider hosting the document database, the company expected to be billed for 26 MB of ingress data.

Why is this?

The problem is how the system updates the state of the device, instead of appending incoming data to the document the complete document is updated (replaced). For one trip this is the amount of data being sent to the document database.

# PackageTotal data Received from device (bytes)Data sent to document database (bytes)
11 4881 488
22 9764 464
34 4647 440
45 95210 416
57 44013 392
68 92816 368
710 41619 344
811 90422 320
913 39225 296
1014 88028 272
1116 36831 248
1217 85634 224
1319 34437 200
1420 83240 176
1522 32043 152
1623 80846 128
1725 29649 104
1826 78452 080
1928 27255 056
2029 76058 032
2131 24861 008
2232 73663 984
2334 22466 960
2435 71269 936
2537 20072 912
2638 68875 888
2740 17678 864
2841 66481 840
2943 15284 816
3044 64087 792

Total data sent to the document database for one trip 1 339 200 bytes

To understand, for the first data package received we send 1 488 bytes to the document database, for the second data package we send 1 488 * 2 = 2 975 bytes plus the 1 488 bytes we sent for the first package and so on.

Expected result would be that we send ~44 Kilobytes, but we send 1.3 Megabytes.

Over time it equates to

Actual Data SizeExpected Data Size
One trip1.3 MB44 KB
One dayOne trip * 20 = 26 MBOne trip * 20 = 0.80 MB
One weekOne day * 07 = 182 MBOne day * 07 = 6.10 MB
One monthOne day * 30 = 780 MBOne day * 30 = 26.4 MB

The system has sent 30 times the expected data size to the document database. With 10 000 devices in the system, it would have sent 7.8 TB to the document database over a month; instead of 0.264 TB (264 GB)

The solution

Instead of using the ReplaceOneAsync method, the system should be changed to use the Push method like so:

await _collection.UpdateOneAsync(s => s.Id == id, Builders<DeviceState?>.Update.Push(d => d.Data, dataPackage), new UpdateOptions { IsUpsert = true });

The push operator appends the data to an array

EF Core – Get/fetch from Stored Procedure

Entity Framework Core 3, Uncategorized

Scenario, fetch a user from the database using a stored procedure.

If the stored procedure does not return an exact type that is already registered in the DbContext, the type has to be added and mapped.

In our scenario the type we are fetching is not mapped.

public class UserFromSproc
{
    public string Email { get; set; }
    /* ... */   
}

Context mapping:

public partial class MyContext : DbContext
{    
    public DbSet<UserFromSproc> UserFromSproc { get; set; }

    /* ... */

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<UserFromSproc>(entity =>
        {
            entity.HasNoKey().ToView(null);
        });
    }
}

The ToView(null) part is important, because if we use a code first approach; it tells the code generator that we should not map this type to table. Hence, no table would be created for our type UserFromSproc.

Executing the stored procedure

var parameter = new SqlParameter("@userId", 
                                 System.Data.SqlDbType.Int).Value = userId;
var user ctx.UserVehiclesWithScope
            .FromSqlRaw($"GetUserById {0}", parameter)
            .SingleOrDefaultAsync();

Delete bound certificate

Programming

If the current certificate is about to expire, or, we have bound a invalid certificate we need to be able to delete the binding. First we will list all bound certificates by executing netsh http show sslcert.

Output

IP:port                      : 0.0.0.0:443
Certificate Hash             : certificate thumbprint
Application ID               : {12345678-db90-4b66-8b01-88f7af2e36bf}
/ ... /
DS Mapper Usage              : Disabled
Negotiate Client Certificate : Disabled
Reject Connections           : Disabled

To delete the binding we will execute delete sslcert ipport=0.0.0.0:443

Binding certificate to a Windows Service

Programming

If your are hosting, for example SignalR using a Windows Service; it is appropriate to only communicate using a secure transport. To enable this a certificate needs to be bound to the port the Windows Service is listening on. In this example we will be communicating using port 443.

To make this happen, we will use Netsh https://docs.microsoft.com/en-us/windows-server/networking/technologies/netsh/netsh-http

The command to execute is:

netsh http add sslcert ipport=0.0.0.0:443 appid={91ae3467-05f7-4eef-9903-017xfc1e72ca} certhash=<your cert thumbprint>

ipport=0.0.0.0:443

0.0.0.0 means it will listen to all IPv4 addresses assigned to the server/machine/computer, so if the server/machine/computer has two IP-addresses 99.99.99.99 and 88.88.88.88 the Windows Service will be reachable at both IP-addresses.

appid={91ae3467-05f7-4eef-9903-017xfc1e72ca}

This is a static constant value, use any valid GUID

certhash=<your cert thumbprint>

The thumbprint of the certificate in use, the thumbprint can be found by right click on a cert-file and selecting properties, select Details tab and scroll down to Thumbprint. External reference/guide: https://onlinehelp.coveo.com/en/ces/7.0/administrator/finding_the_thumbprint_of_a_certificate.htm

Navigation properties can only participate in a single relationship

Entity Framework Core 3, Programming

I had these two Entity Framework classes:

public class Customer
{
    public int CustomerId { get; set; }
    public int BillingAddressId { get; set; }
    public int ShippingAddressId { get; set; }

    public virtual Address BillingAddress { get; set; }
    public virtual Address ShippingAddress { get; set; }

    /* Omitted */
}

public class Address
{
    public int AddressId { get; set; }
    /* Omitted */
    public virtual ICollection<Customer> Customer { get; set; }
}

I had the following mapping

modelBuilder.Entity<Customer>(entity =>
{    
    entity.HasOne(d => d.BillingAddress)
        .WithMany(p => p.Customer)
        .HasForeignKey(d => d.BillingAddressId);

    entity.HasOne(d => d.ShippingAddress)
        .WithMany(p => p.Customer)
        .HasForeignKey(d => d.ShipmentAddressId);
});

I was running the program and got the following exception

System.InvalidOperationException: ‘Cannot create a relationship between ‘Address.Customer’ and ‘Customer.ShippingAddress’, because there already is a relationship between ‘Address.Customer’ and ‘Customer.BillingAddress’. Navigation properties can only participate in a single relationship.

The exception is pretty clear, but still it took some thinking to understand the issue. What’s need is a second property for the second relationship, hence, we end up with the following models

public class Customer
{
    public int CustomerId { get; set; }
    public int BillingAddressId { get; set; }
    public int ShippingAddressId { get; set; }

    public virtual Address BillingAddress { get; set; }
    public virtual Address ShippingAddress { get; set; }

    /* Omitted */
}

public class Address
{
    public int AddressId { get; set; }
    /* Omitted */
    public virtual ICollection<Customer> CustomerBillingAddresses { get; set; }
    public virtual ICollection<Customer> CustomerShippingAddresses { get; set; }
}

Note that we have created another property for the second relationship, and renamed the first relationship. The mapping was changed to

modelBuilder.Entity<Customer>(entity =>
{    
    entity.HasOne(d => d.BillingAddress)
        .WithMany(p => p.CustomerBillingAddresses )
        .HasForeignKey(d => d.BillingAddressId);

    entity.HasOne(d => d.ShippingAddress)
        .WithMany(p => p.CustomerShippingAddresses )
        .HasForeignKey(d => d.CustomerShippingAddresses );
});

Bingo, after the above changes the program executed as expected

Error when trying to delete entity in Entity Framework 4

Programming

I was getting an error message in Entity Framework 4 when I was trying to delete an entity.
The entity had a relationship to another entity, so when I tried to delete the parent entity my child entity would have had a reference to an entity that didn’t exist.

What I want is when I delete the top entity the child entity should also be deleted; this is called cascade delete.

This webpage describes the mechanics behind it, it worked for me and is a great write-up:
http://blogs.msdn.com/b/alexj/archive/2009/08/19/tip-33-how-cascade-delete-really-works-in-ef.aspx

Unable to update the entityset because it has a definingquery and no insertfunction

Programming

I was trying to insert a record into a table using Entity Framework 4 (EF 4) and got this error message back from the data store.

I did a couple of bing and google queries and found the following links:
http://stackoverflow.com/questions/1589166/it-has-a-definingquery-but-no-insertfunction-element-err

Although it seemed strange that I received this error message now, I have never every before had any issues with inserting records.

I started to look at the table I was trying to insert records to and I found that the table was missing a primary key.
After adding the primary key, I did not receive this message again.

Android Creating Notification Icons, Menu Icons, Launcher icons, Menu icons, Action bar icons (Android 3.0+), Tab icons; Android Asset Studio

Programming

A great tool to create a set of different icons, it is called Android Asset Studio:
http://android-ui-utils.googlecode.com/hg/asset-studio/dist/index.html

A great tool to create

  1. Launcher icons
  2. Menu icons
  3. Action bar icons (Android 3.0+)
  4. Tab icons
  5. Notification icons

It works best in Google Chrome

ASP.NET MVC 3 Unable to install using Microsoft Web Platform Installer

Programming

After watching some videos from PDC, MIX 11 and Tech-Ed 2011 I wanted to install ASP.NET MVC 3.
This should now be very easy with the Web Platform Installer, well it wasn’t.

It appears as though there are some registry changes made when you install Visual Studio 2010 SP 1 thats makes it impossible to install ASP.NET MVC 3.

The cause of the problem is that Microsoft ASP.NET Web Pages is installed and the uninstaller is unable to to uninstall the old version and install a new version of ASP.NET Web Pages.

Fortunately there is a solution to the problem, four steps:

  1. Remove the trailing backslash from the following registry keys:

    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET\4.0.30319.0\Path

    HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\ASP.NET\4.0.30319.0\Path

  2. Uninstall the old version of Microsoft ASP.NET Web Pages.
  3. Add the trailing backslash back to those keys.
  4. Install MVC 3


That worked for me, does it work for you?
Source: http://connect.microsoft.com/VisualStudio/feedback/details/650729/can-not-install-mvc-3

Android an easier way to test that a service that should start at device boot; starts

Programming

Earlier I wrote a blog post about how to test that an Android service starts at device boot. Well there is an easier way to start a virtual device.

Start the Android SDK and AVD Manager, it is located in your Windows Android SDK folder. The file name is SDK Manager.exe. Select Virtual devices.

image

Once you have started the SDK and AVD Manager, select the virtual device you used to test your application/service from your IDE (Eclipse, IntelliJ); press Start… If you have programmed your application and service correctly; the service should now start when you start your virtual device.

Android How to test that a service that should start at device boot; starts

Programming

Developing Android services that should start when the phone is started (after boot-up) is very easy and is very convenient for the user.

It is especially convenient for the user, because, if the service sits in the background and does its thing with minimal user interaction; it could be hard for the user to remember to start the service when the phone is restarted.

Here is a tutorial on how to get the functionality of a service starting at device boot:
http://blog.gregfiumara.com/archives/82

Well how do we actually test in the emulator that it works?

  1. Develop your android activity and service using the tutorial above
  2. Compile the application and start the emulator from you favorite IDE (Eclipse or IntelliJ)
  3. Make sure the application can start the service

After these steps the application is stored on the emulator and the service is registered to start when the device boot. What we have to do is to start the emulator but no inside of our IDE (Eclipse or IntelliJ)

How do we do this?

  1. Locate where you installed your Android SDK (in my case C:\Program Files\Android\android-sdk-windows)
  2. Open the tools directory within the Android SDK folder, in this folder there should be a file called emulator.exe
  3. Start the command promt; Start Meny -> Run -> cmd.exe
  4. Navigate to the tools directory in step 2, command: cd C:\Program Files (x86)\Android\android-sdk-windows\tools
  5. Before you start the emulator you have to decide which emulator to start. I have two emulators named AAVD1, AAVD2. You want to start the same emulator that was launched when you launched the emulator from your IDE. In my case it was AAVD2.

From the command prompt you should now start the emulator with this command:
C:\Program Files\Android\android-sdk-windows\tools>emulator -avd AAVD2

Now you should see your service start at device boot.

You could also start you emulator from the SDK Manager (SDK Manager.exe) which i located in the Android SDK directory.

WCF Duplex Service Deadlock

Programming

I was creating a WCF duplex service and was experiencing some weird behavior.
The service was sending messages to the client (duplex) but as soon the client (any client) wanted to unsubscribe to the service, hence, stop recieving messages; the service and all clients would freeze.

After some binging and googling I found a solution, it seems that there is such a thing as Synchronization Context; by setting this attribute to false all problems solved.

You need to set this attribute on your class implementing the WCF service callback

[CallbackBehavior(UseSynchronizationContext = false)]
public partial class Form1 : Form, IMyServiceCallback
{
	IServiceService proxy;
	public Form1()
	{
		InitializeComponent();
	}
}

More in-depth information provided in these articles:
http://www.aaronmurrell.com/ThinkBlog/WCFThreadSynchronizationContext.aspx
http://msdn.microsoft.com/en-us/magazine/cc163321.aspx

Android Split Horizontal Tablelayout 50 % width

Programming

I was trying to figure out how to create an Android view with a tablelayout that was split in half:
|control| control|
|control | control|

It took some time, but i figured out it is pretty simple; the key is to use a linearlayout with orientation horizontal:

<?xml version="1.0" encoding="utf-8"?>
<ScrollView 
	xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/scroller"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:fillViewport="true" >
	<LinearLayout 
    android:orientation="vertical"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent">
    	
    	<LinearLayout 
	    android:orientation="horizontal"
	    android:layout_width="fill_parent"
	    android:layout_height="wrap_content">
			<LinearLayout 	
				android:layout_weight="1" 
				android:layout_height="fill_parent" 
				android:layout_width="fill_parent" 
				android:orientation="vertical">
				<TextView 
					android:text="User Applications" 
					android:layout_width="wrap_content" 
					android:layout_height="wrap_content" />
				<TextView 
					android:id="@+id/tvUserApplications" 
					android:layout_width="wrap_content" 
					android:layout_height="wrap_content" />
			</LinearLayout>
			
			<LinearLayout 
				android:layout_weight="1" 
				android:layout_height="fill_parent" 
				android:layout_width="fill_parent" 
				android:orientation="vertical">
				<TextView 
					android:text="System Applications" 
					android:layout_width="fill_parent" 
					android:layout_height="wrap_content" />
				<TextView 
					android:id="@+id/tvSystemApplications" 
					android:layout_width="wrap_content" 
					android:layout_height="wrap_content" />
			</LinearLayout>
		</LinearLayout>
    </LinearLayout>   
</ScrollView>

This code snippet will create a view with tablelayout with two rows split in two (two columns) each having 50 % width with a textview inside of each column.

The key point is to define android:layout_weight=”1″ for each LinearLayout.

Happy coding.

Silverlight 3, Create Unit Tests, Empty Dialog

Programming

I had this problem a couple of week ago.
I had a solution with a couple of projects, one was a Silverlight 3 project, and the one I was creating the unit test for was a database layer.

Well, when I right mouse button clicked on one of my database layer’s functions and selected “Create Unit Tests” the “Create Unit Tests” dialog is all empty.

Like so:

Create Unit Tests Dialog

Create Unit Tests Dialog

It’s seems to be a bug when you have a Silverlight 3 project and you try to create a unit test for another project in the solution.
Microsoft even has a KB for it:
http://code.msdn.microsoft.com/KB962866

There seems to be two solutions for the problem either unload your Silverlight project or install the hot fix mentioned in the KB.

WCF 3.5 Shared hosting

Programming

This seems like a fairly common problem.
You test your WCF 3.5 service on your own IIS or in the virtual development server and all seems to work fine; then you deploy your project to your hosting site.

If this site is shared hosting; that is: you upload your wfc, aspx, silverlight projects to the “cloud” it will mos def cause problems.
Well this is a solution that worked with my shared hosting site, it all involves changes and additions to the config-file.

1. Add a baseAddressPrefixFilters to the system.serviceModel tag

<system.serviceModel>
      <serviceHostingEnvironment>
        <baseAddressPrefixFilters>
          <add prefix="http://cm.f3r.se/"/>
        </baseAddressPrefixFilters>
      </serviceHostingEnvironment>

2. Add a baseAddress to your serive tag

<service name="my.service.name" behaviorConfiguration="my.behavior.configuration">
        <endpoint address="http://my.endpointaddress.com/myservice.svc/" binding="webHttpBinding" contract=my.serivce.interface">
        </endpoint>
        <host>
          <baseAddresses>
            <add baseAddress="http://www.myservice.com/" />
          </baseAddresses>
        </host>
      </service>

The endpoint address HAS TO BE the complete url to your serivce.

It work for me, does it work for you?

The specified named connection is either not found in the configuration, not intended to be used with the EntityClient provider, or not valid.

Programming

I was getting this error message in my database layer when I was trying to access some of my data using Entity Framework 4.

Well since my solution contains several projects, one for the GUI (the executable, .exe) and on for the data access using EF4, and some more.

The problem is that you HAVE to specify the connection string(s) in the app.config project where you have the executable.

In my case I only had defined the connection string(s) in my data access layer and not in my GUI layer.

Write special characters to file (skriva svenska tecken)

Programming

Earlier I was talking about creating CSV-files with the current cultures list separator.
Well I stumbled upon a new problem, when I was writing special characters (Swedish characters such as å, ä and ö) they ended up looking all shaggy in the file.

My solution was to change the ContentEncoding:

HttpContext.Current.Response.ContentEncoding = System.Text.Encoding.Unicode;

Now it works.

WCF, Service Trace Viewer, maxItemsInObjectGraph

Programming

I was working with a WCF 4 service today that is supposed to send quit a lot of data over the internal network. The service supports two binding netTcpBinding and wsHttpBindnig.

Calling the service worked out fine when dealing with small chunks of the data, but when trying to retrieve all of the data I kept getting these strange exceptions (on the client side). I got different exceptions for the two bindings.

wsHttpBindingError

wsHttpBinding exception

netTcpBindingError

netTcpBinding exception

On the client and server side I had turned up all options I could think of such as:

maxReceivedMessageSize
maxBufferPoolSize
maxArrayLength
maxStringContentLength

I had heard of Service Trace Viewer before but never really used it, and that’s a shame because it really useful when trying to figure out whats wrong with a WCF service.

I enabled tracing in my service.webconfig file:

<system.diagnostics>
  <trace autoflush="true" />
  <sources>
    <source name="System.ServiceModel"
                  switchValue="Information, ActivityTracing"
                  propagateActivity="true">
      <listeners>
        <add name="sdt"
            type="System.Diagnostics.XmlWriterTraceListener"
            initializeData= "c:\SdrConfig.e2e" />
      </listeners>
    </source>
    <source name="System.ServiceModel.MessageLogging">
      <listeners>
        <add name="messages"
        type="System.Diagnostics.XmlWriterTraceListener"
        initializeData="c:\messages.svclog" />
      </listeners>
    </source>
  </sources>
</system.diagnostics>
<system.serviceModel>

<diagnostics>
  <messageLogging
      logEntireMessage="true"
      logMalformedMessages="true"
      logMessagesAtServiceLevel="true"
      logMessagesAtTransportLevel="true"
      maxMessagesToLog="3000"
      maxSizeOfMessageToLog="2000"/>
</diagnostics>

I repeated the steps to reproduce my error and fired up Service Trace Viewer and opened my trace-file.

Boom goes the dynamite, there it was:

There was an error while trying to serialize parameter http://tempuri.org/:MyFcuntion. The InnerException message was ‘Maximum number of items that can be serialized or deserialized in an object graph is ‘65536’. Change the object graph or increase the MaxItemsInObjectGraph quota.

So there was yet another max-setting i hadn’t configured.
I added the bold part to my service web.config-file

<behavior name="ServiceBehaivor">
  <serviceMetadata httpGetEnabled="true"/>
  <serviceDebug includeExceptionDetailInFaults="true"/>
  <serviceThrottling maxConcurrentCalls="2147483647" />
  <dataContractSerializer maxItemsInObjectGraph="2147483647" />
</behavior>

I added the following endpoint behavior to my app.config-file

<behaviors>
  <endpointBehaviors>
    <behavior name="Behaviors.EndpointBehavior">
      <dataContractSerializer maxItemsInObjectGraph="2147483647" />
     </behavior>
  </endpointBehaviors>
</behaviors>

It took me some hours to figure this out, if I would have used the Service Trace Viewer from the get go I mos def could have found the solution a lot quicker. Live and learn.

Generate CSV, all columns in one cell

Programming

I had to generate a CSV-file from a list of people. Creating the file with a header-row and filling it with data was not a problem. The problem occurred when i opened the file in Excel. All the data ended up in one cell, cell A.

I separated the file with commas (of course, CSV = comma-separated-values).

I did some research and it appears that Windows has a setting that is called list separator, Region and Language -> Additional Settings -> List separator (Windows 7).

I found out that my Windows installation is using ; as a list separator.

After changing my function that generate the CSV-file to use ; as the delimiter Excel opens and displays the file correctly with each column in a separate cell.

But since every client could possibly have their own list separator I used the following code to figure out which list separator to use:

using System.Globalization;
CultureInfo.CurrentCulture.TextInfo.ListSeparator;