Yesterday

Quino v1.13.0: Schema migration, remoting, services and web apps

Published by marco on in Programming

The summary below describes major new features, items of note and breaking changes in Quino. The full list of issues is also available for those with access to the Encodo issue tracker.

Highlights

Data & Schema

Remoting & services

  • Fixed several issues in the remoting driver (client and server parts). (QNO-4626, QNO-4630, QNO-4631, QNO-4388, QNO-4575, QNO-4629, QNO-4573, QNO-4625, QNO-4633, QNO-4575)
  • Added a runner for Windows services that allows debugging and shows logging output for applications that use the CoreServiceBase, which extends the standard .NET ServiceBase. The runner is available in the Encodo.Service assembly.

Web

  • Improved default and custom authentication in web applications and the remoting server. Also improved support for authorization for remote-method routes as well as MVC controllers.
  • Improved configuration, error-handling and stability of the HttpApplicationBase, especially in situations where the application fails to start. Error-page handling was also improved, including handling for Windows Event Log errors.
  • Improved appearance of the web-based schema migrator. (QNO-4559, QNO-4561, QNO-4563, QNO-4548, QNO-4487, QNO-4486, QNO-4488)

Winform

  • Data-provider statistics: improved the WinForm-based statistics form. (QNO-4231, QNO-4545, QNO-4546)
  • Standard forms: updated the standard WinForm about window and splash screen to use Encodo web-site CI. (QNO-4529)

System & Tools

  • Removed the dependency on the SmartWeakEvents library from Quino. (QNO-4645); the Quino and Encodo assemblies now no longer have any external dependencies.
  • Image handling: the Encodo and Quino libraries now use the Windows Imaging Components instead of System.Drawing. (QNO-4536)
  • Window 8.1: fixed culture-handling for en-US and de-CH that is broken in Windows 8.1. (QNO-4534, QNO-4553)
  • R# annotations have been added to the Encodo assembly. Tell R# to look in the Encodo.Core namespace to use annotations like NotNull and CanBeNull with parameters and results. (QNO-4508)
  • Generated code now includes a property that returns a ValueListObject for each enum property in the metadata. For example, for a property named State of type CoreState, the generated code includes the former properties for the enum and the foreign key backing it, but now also includes the ValueListObject property. This new property provides easy access to the captions.

    public CoreState State { … }
    public ValueListObject StateObject { … }
    public int? CoreStateIdId { … }
  • Improved the nant fix command in the default build tools to fix the assembly name as well. The build tools are available in bin/tools/build. See the src/demo/Demo.build file for an example on how to use the Nant build scripts for your own solutions. To change the company name used by the “fix” command, for example, add the following task override:

    <target name="fix.before">
      <call target="fix.before.base"/>
      <property name="InfoCompanyName" value="Foobar Corporation"/>
    </target>
  • Fixed the implementation of IntegrateRemotableMethods to avoid a race condition with remote methods. Also improved the stability of the DataProvider statistics. (QNO-4599)

Breaking changes

  • The generic argument TRight has been removed from all classes and interfaces in the Encodo.Security.* namespace. In order to fix this code, just remove the int generic parameter wherever it was used. For example, where before you used the interface IUser<int>, you should now use IUser (QNO-4576).
  • The overridable method MetaAccessControl.DoGetAccessChecker() has been renamed to MetaAccessControl.GetAccessChecker().
  • Renamed the Encodo.ServiceLocator.SimpleInjector.dll to Encodo.Services.SimpleInjector.dll and Quino.ServiceLocator.SimpleInjector.dll to Quino.Services.SimpleInjector.dll Also changed the namespace Quino.ServiceLocator to Encodo.Quino.Services.
  • Renamed HttpApplicationBase.StartMetaApplication() to CreateAndStartUpApplication().
  • Classes may no longer contain properties with names that conflict with properties of IMetaReadable (e.g. Deleted, Persisted). The model will no longer validate until the properties have been renamed and the code regenerated. (QNO-4185)
  • Removed StandardIntRights with integer constants and replaced it with StandardRights with string constants.
  • The IAccessControl.Check() and other related methods now accept a sequence of string rights rather than integers.
  • IMetaConfiguration.ConfigureSession() has been deprecated. The method will still be called but may have undesired side-effects, depending on why it was overridden. The common use was to initialize a custom AccessControl for the session. Continuing to do so may overwrite the current user set by the default Winform startup. Instead, applications should use the IDataSessionAccessControlFactory and IDataSessionFactory to customize the data sessions and access controls returned for an application. In order to attach an access control, take care to only set your custom access control for sessions that correspond to your application model.[1]

    internal class JobVortexDataSessionAccessControlFactory : DataSessionAccessControlFactory
    {
      public override IAccessControl CreateAccessControl(IDataSession session)
      {
        if (session.Application.Model.MetaId == JobVortexModelGenerator.ModelGuid)
        {
          return new JobVortexAccessControl(session);
        }
    
        return base.CreateAccessControl(session);
      }
    }
  • The default length of the UserModule.User.PasswordHash property has been increased from 100 characters to 1000. This default is more sensible for implementations that use much longer validations tokens instead of passwords. To avoid the schema migration, revert the change by setting the property default length back to 0 in your application model, after importing the security module, as shown below.

    var securityModule = Builder.Include<SecurityModuleGenerator>();      
    securityModule.Elements.Classes.User.Properties[
      Encodo.Quino.Models.Security.Classes.SecurityUser.Fields.PasswordHash
    ].MaximumSize = 100;
  • Application.Credentials has been removed. To fix references, retrieve the IUserCredentialsManager from the service locator. For example, the following code returns the current user:

    Session.Application.Configuration.ServiceLocator.GetInstance<IUserCredentialsManager>().Current

    If your application uses the WinformMetaConfigurationTools.IntegrateWinformPackages() or WinformDxMetaConfigurationTools.IntegrateWinformDxPackages(), then the IDataSession.AccessControl.CurrentUser will continue to be set correctly. If not, add the SingleUserApplicationConfigurationPackage to your application’s configuration. The user in the remoting server will be set up correctly. Add the WebApplicationConfigurationPackage to web applications in order to ensure that the current user is set up correctly for each request. (QNO-4596)

  • IDataSession.SyncRoot has been removed as it was no longer needed or used in Quino itself. Sessions should not be used in multiple threads, so there is no need for a SyncRoot. Code that uses it should be reworked to use a separate session for each thread.
  • Moved IMetaApplication.CreateSession() to an extension method. Add Encodo.Quino.App to the using clauses to fix any compile errors.
  • Removed IMetaApplication.DataProvider; use IMetaApplication.Configuration.DataProvider instead. (QNO-4604)
  • The schema migration API has been completely overhauled. ISchemaChange and descendents has been completely removed. ISchemaAction is no longer part of the external API, although it is still used internally. The ISchemaChangeFactory has been renamed to ISchemaCommandFactory and, instead of creating change objects, which are then applied directly, returns ISchemaCommand objects, which can be either executed or transformed in some other way. IMigrateToolkit.GetActionFor() has also been replace with CreateCommands(), which mirrors the rest of the API by returning a sequence of commands to address a given ISchemaDifference. This release still has some commands that cannot be transformed to pure SQL, but the goal is to be able to generate pure SQL for a schema migration. (QNO-993, QNO-4579, QNO-4581, 4588, 4591, QNO-4594)
  • IMigrateSchemaAspect.Apply() has been removed. All aspects will have to be updated to implement GetCommands() instead, or to use one of the available base classes, like UpdateDataAspectBase or ConvertPropertyTypeSchemaAspect. The following example shows how to use the UpdateDataAspectBase to customize migration for a renamed property.

    internal class ArchivedMigrationAspect : UpdateDataAspectBase
    {
      public ArchivedMigrationAspect()
        : base("ArchivedMigrationAspect", DifferenceType.RenamedProperty, ChangePhase.Instead)
      {
      }
    
      protected override void UpdateData(IMigrateContext context, ISchemaDifference difference)
      {
        using (var session = context.CreateSession(difference))
        {
          session.ChangeAndSaveAll<Project>(UpdateArchivedFlag);
        }
      }
    
      private void UpdateArchivedFlag(Project obj)
      {
        obj.Archived = !obj.Archived;
      }
    }

    The base aspects should cover most needs; if your functionality is completely customized, you can easily pass your previous implementation of Apply() to a DelegateSchemaCommand and return that from your implementation of GetCommands(). See the implementation of UpdateDataAspectBase for more examples. (QNO-4580)

  • MetaObjectIdEqualityComparer<T> can no longer be constructed directly. Instead, use MetaObjectIdEqualityComparer<Project>.Default.
  • Renamed MetaClipboardControlDx.UpdateColorSkinaware() to MetaClipboardControlDx.UpdateSkinAwareColors().
  • IMetaUnique.LogicalParent has been moved to IMetaBase. Since IMetaUnique inherits from IMetaBase, it is unlikely that code is affected (unless reflection or some other direct means was used to reference the property). (QNO-4586)
  • IUntypedMessage has been removed; the AssociatedObject formerly found there has been moved to IMessage.
  • ITypedMessage.AssociatedObject has been renamed to ITypedMessage.TypedAssociatedObject. (QNO-4647)
  • Renamed MetaObjectTools to MetaReadableTools.
  • Redefined the protected methods GenericObject.GetAsGuid() and GenericObject.GetAsGuidDefault as extension methods in MetaWritableTools.
  • IMetaFeedback.CreateGlobalContext() has been removed. Instead the IGlobalContext is created using the service locator.


[1] The schema migration creates a metadata model for your model—meta-metadata—and uses the Quino ORM to load data when importing a model from a database. If you aren’t careful, as shown in the code example, then you’ll attach your custom access control to the sessions created for the schema migration’s data-access, which will more than likely fail when it tries to load user data from a table that does not exist in that model.

4 days Ago

Ukraine Update

Published by marco on in Public Policy & Politics

The article The IMF’s New Cold War Loan to Ukraine by Michael Hudson (CounterPunch) provides some interesting insight into the IMF’s machinations on behalf of its masters in Europe and the U.S.

“[…] the IMF signed off on the first loan ever to a side engaged in a civil war, not to mention rife with insider capital flight and a collapsing balance of payments.”

The IMF has hard and fast rules for loaning money and is famous the world over for being an exceedingly unforgiving creditor…unless the creditor is the European continent’s newest democracy, the propping up of which is an opportunity to provoke the Russian bear that is just too good to pass up.

“Based on fictitiously trouble-free projections of the ability to pay, the loan supported Ukraine’s hernia currency long enough to enable the oligarchs’ banks to move their money quickly into Western hard-currency accounts before the hernia plunged further and was worth even fewer euros and dollars.”

Not only does propping up Ukraine’s currency help keep that civil war alive—and keep Russia occupied—but it also—and this is purely coincidental, mind you—provides enough cash for all of the usual exceedingly rich suspects to continue being exceedingly rich. This by the usual mechanism of roping the collective taxpayers of the world into paying for said rich individuals’ bad investments by their respective governments, most of which are also comprised of rich people or those who would like to be rich people or who are otherwise beholden to them.

“In practice, the IMF simply advances however much a government needs to bail out its bankers and bondholders, pretending that more austerity enhances the ability to pay, not worsen it.”

As the rulers of the world have realized that people are much more interested in bad housewives, cute cat pictures and an explosion of inanity on social media (first-world version) or are utterly distracted by the sheer misery that is every waking hour of their lives (third-world version), they make less and less of an effort to hide how they’re ripping everyone off.

“Ukraine looks like a replay of the Greek situation with an exclamation mark! One official last year called its Debt Sustainability Analysis, “‘a joke,’ a [European] commission official described it ‘a fairy tale to put children to sleep’ and a Greek finance ministry official said it was ‘scientifically ridiculous.’””

I mean, why even bother to expend any effort hiding what’s going on when (A) very few are paying any attention at all and (B) no-one can really do anything about it. And the analysis that we do get on issues of merit are skewed by the authors’ paymasters. Here, Hudson describes the obvious and reprehensible cherry-picking of history that an ostensibly serious analysis engaged in when she tried to declare some debt as “odious”.

“The double standard here is that instead of labeling Ukraine’s entire series of post-1991 kleptocratic governments odious, she singles out only Yanukovich, as if his predecessors and successors are not equally venal. But an even greater danger in trying to declare Ukraine’s debt “odious”: It may backfire on the United States, given its own support for military dictatorships and kleptocracies.”

The designation “odious debt” carries with it a freight of baggage. As Hudson points out, the U.S. has instilled no small amount of such debt in its various puppets over the decades. This is, however, an argument that concerns Hudson, but not the original author he attempts to chastise. As they say, “if you have no taste, you can do anything”. A useful corollary for today would be: “If you don’t care about consistency or hypocrisy on your own part, you can write throw as many stones as you like”.

The article Gaza and the Threat of World War by John Pilger (CounterPunch) contained another likely futile attempt to point out the parallels to even very recent debacles of failed diplomacy and deceitful motives: the most recent invasion of Iraq. Though the U.S. media has seemingly formed a diamond of fact out of the coal dust of rumors of Russian invasions and troop involvements, there are others who are a bit more gun-shy.

“What matters is a Russian “invasion” of Ukraine that seems difficult to prove beyond familiar satellite images that evoke Colin Powell’s fictional presentation to the United Nations “proving” that Saddam Hussein had WMD. “You need to know that accusations of a major Russian ‘invasion’ of Ukraine appear not to be supported by reliable intelligence,” wrote a group of former senior US intelligence officials and analysts, the Veteran Intelligence Professionals for Sanity, to German Chancellor Angela Merkel. “Rather, the ‘intelligence’ seems to be of the same dubious, politically ‘fixed’ kind used 12 years ago to ‘justify’ the U.S.-led attack on Iraq.””

But what of those who are perhaps not so stupid, who make what, on the surface, seem to be cogent arguments in support of a free Ukraine and a liberated Russian people, free from the yoke of Putin? The article Back to Yalta? Stephen Cohen and the Ukrainian crisis by Nikolay Koposov (Eurozine) is one such article. In this article, Koposov takes on Stephen Cohen, painted as a Putin-lover but who I’ve found to be relatively balanced and coherent in the few interviews I’ve heard. Anyway, let’s let Koposov describe his thesis,

“For his part, Cohen presents himself as a “political realist” and an American patriot whose concern is the security of the United States, which according to him has been consistently undermined by US policymakers and experts whose incompetence and “Putinophobic follies” have deprived the United States of “the best potential partner we had anywhere in the world to pursue our national security”.”

I fail so far to see the problem with Cohen’s positions as described in this article. The conclusion implied by the tone is unsupported by the evidence provided. That the tone is so snide, disparaging and dismissive indicates strongly that the author will not be as even-handed as one might hope or as his introduction suggests. He seems to see everything as black and white, with poor Stephen Cohen seemingly incapable of doing anything right.

If you don’t support the U.S. in its crusade, you’re a Putin supporter. The fact is that the U.S. has provably been a force for evil in the world—for pretty much everyone other than a small handful of the privileged. It has been an overall detriment for the people it bombs and for its own citizens, who are constantly short-changed by its policies. The changes proposed by the U.S. have come to tears for many other countries, Iraq being only the most recent and prominent victim. Afghanistan, Libya and Palestine may also have something to add about the effectiveness of U.S. policy for improving their countries.

Should the U.S. get its way in Ukraine, there is every reason to believe—simply by looking at the history books, which are clear—that conditions will be worse for Russians than even the awfulness of the reality of today’s kleptocracy. Said state of affairs being something for which the U.S. and its allies are largely responsible. It’s not a coincidence that the rich and powerful did so well in the post-Glasnost years.

“Cohen blames Bill Clinton for beginning “NATO’s eastward expansion”, which hurt Russia. Is he suggesting Clinton’s only legitimate concern should have been Russia’s security, as opposed to say Estonia’s? Did Russians occupy Estonia in 1940, or did Estonians occupy Russia? Who has better grounds for feeling insecure? Or is it the case that only nuclear giants may have legitimate concerns about their national security? ”

Now I’m thinking that this guy is a facile idiot who seems to honestly believe that NATO has Estonia’s concerns at heart. It’s not that only nuclear giants can express their concerns, but that for the last twenty years we have heard only one voice: that of the U.S. Even when Estonia talks, it’s either the U.S. or—hardly better—the EU that makes its mouth move. The U.S. and the EU have shown themselves to be self-interested to the core. They have every interest in portraying Russia as an intractable enemy and giving it the Libya treatment, if possible.

That is the dream, anyway. If they could finally topple the Russian empire, Europe would finally have its resource problem under control, or would at least have cozier terms with a Russian territory under U.S. control. Let’s not lie to ourselves, this is the dream. The U.S. doesn’t give two shits for democracy in Estonia. It wants to build missile bases ever closer to its age-old enemy, the only country capable of resisting it with nukes. China would be logically next.

Don’t think that this is not the official policy of the U.S. It’s probably even written down in no uncertain terms somewhere. NATO is a tool to be used to enact this vision; it is not there to protect European countries. To believe so is utter foolishness.

Even the Yeltsin years, during which Russians had hope, despite the torrent of capital and power rushing out of their country at the time, would likely be a paradise compared to what is to come, should the U.S. get its way. But hope dies last, as the Germans like to say, and the author is eminently hopeful to lend the U.S. support in the hopes that this time it will be different.

“For all Yeltsin’s imperfections, Russia had a moment of relative freedom (especially freedom of expression) in the 1990s, which did not end until the formation of Putin’s regime. In a similar way, conflicts between Yeltsin’s Russia and its eastern European partners over the interpretation of history were rare.”

Bullshit. There was never a democracy in anything but name in Russia. No more so than the purported democracy the U.S. enjoys in this, the second gilded age, the age of plutocrats. The Russian government was allowed to play at democracy like a little girl with her teddy bears plays tea party. Meanwhile, the country was stolen out from under the Russian people by rapacious so-called free-market capitalists, who differed in no way from the Huns or Ghengis Khan in their ruthlessness. With the approval of a handful of Apparatchiks who also benefitted, armies of lawyers and financiers dismantled the Soviet Union rather more efficiently than an invading army could have.

The Russians went from a totalitarian occupation to an economic one. As mentioned at the top of this essay, the IMF is moving full steam ahead to do the same for Ukraine.

Koposov goes on to describe the evils of Russian policy,

“The program of cultural conservatism includes a quest for stability and hatred of change, especially revolutions; an emphasis on traditional values and an alliance with the Orthodox Church; the revival of a Soviet-style anti-intellectualism and a crusade against “deviant behaviour” (including what is called “non-traditional” sexual behaviour). This politics is complemented by repression and new legislation that has considerably increased police control over Russian society.”

Is he writing about the U.S. or Russia here? Do people like the author have no sense of irony? No self-reflection whatsoever? Do they really see things only from one side, without any notion that their depiction of the world as black and white might be a touch simplistic? Anti-intellectualism, repression, gulags—doesn’t that remind you of anything? How can you piss on Russia when Guantánamo is still open? They’re both bad. But to promote the lily-white U.S. vision over the obviously evil Russian one is to lack all nuance and perspective.

This lack of perspective colors everything. Even something relatively glaringly obvious like Crimea is described thusly:

“In a very profound sense, the annexation of Crimea is also an expression of cultural conservatism, with its pre-modern land-hunger and predilection for tangible symbols of power.”

How can he fail to mention or address Sebastopol, one of Russia’s largest naval bases. Can he not even bring himself to mention that this might be the most obvious reason that Russia would want to secure Crimea? It’s not an honorable reason, but one understandable by the realities of a war-suffused world with belligerent entities crawling the earth, looking to pick a fight.

Even in describing the Ukrainian culture, he given the current ruling party the benefit of a doubt that they have not earned, not even if we ignore all of the accusations of fascism and anti-Semitism.

“What has for decades been crucial about Ukraine is that most Ukrainians acknowledged their differences, but wanted to live together. This was a structural foundation of Ukrainian democracy (and one that made Ukraine so different from “monocentric” Russia but similar to countries such as Canada and the UK, among others).”

If most Ukrainians share these feelings of commonality, why do they put up with a new parliament whose first act was to vote to outlaw the Russian language for official purposes? When more than half of the country speaks that language as their primary language? Is this the kind of democratic inclusiveness that he’s talking about? Did this fool of an author really just compare the the newborn civil-war–torn hemi-democracy/hemi-putsch–governed Ukraine to Canada? How can anyone take this seriously?

And then Koposov trots out the old saw that he’s simply trying to get Russia to help itself by trying to keep some friends rather than making everyone an enemy.

“Let me ask Cohen: is it in Russia’s best interests to be a country without friends, except for a couple of other dictatorships?”

In honesty, Russia has friends, but they lie to the east and south rather than the west and north. With the U.S. bending all of its might—both miltary and fiscal—to ensure this outcome, what can Russia realistically do? It can strengthen its ties in Asia, I suppose, for which it will also be castigated. The only solution that will satisfy is a complete capitulation to western desires.

“However, Russia has to accept the right of eastern European countries to be suspicious of it and avoid making them choose between Russia and the West. Whatever other countries’ misdeeds may be, Russia bears the lion’s share of historical responsibility for the issues it has with its neighbours, simply because they were ruled from Moscow, and Russia was not ruled from Kiev, Tallinn or Warsaw. Russia has an obligation to take the lead in peacefully overcoming these issues. I think this would be the only democratic way of defining Russia’s national interests “on its own borders”. President Yeltsin’s policy was at least for a while based upon this understanding. (Emphasis added.) ”

While he’s right that the only thing Russia can do is work with what it’s got, that little emphasized sentence absolves the U.S. and Britain and others of their entire involvement at the geopolitical level, in general, and in Russia and Eastern Europe, in particular.

Russia should, in effect, do what its armed-to-the-teeth NATO neighbors want. Make nice, in other words, do what the U.S. wants—expressed through its proxies.

“Stephen Cohen complains: “If Russia under Yeltsin was presented [by US media] as having legitimate […] national interests, we are now made to believe that Putin’s Russia has none at all.” But this is fairly normal: democracies often cannot accept as legitimate what an authoritarian regime views as its rights.”

What is wrong with that complaint, if one doesn’t accept a priori that Putin is evil incarnate? Or that Cohen is a shill for a new totalitarianism? That even Putin should get the same treatment as other leaders of his country is the heart of diplomacy. But diplomacy is a lost art.

Putin’s concerns cannot, by definition, be legitimate because he’s a dictator—and not one of the good ones because he doesn’t agree with the U.S. unilaterally. It has always been this way: we only keep the ones we like and we only like the ones who suck up. It’s Putin’s own fault for being so intractable. If, however, the Russians choose a different president, they better not choose one that does anything for them, else he’ll follow in the footsteps of Mossadegh or Lumumba or those poor fools in Palestine who voted Hamas into power in a democratic election. Better to choose a manipulable puppet and see if you can get something for yourself.

And to cap things off,

Russian aggression against Ukraine has betrayed the general expectation of the major powers to pursue a responsible and, therefore, predictable politics, such that the world is spared major military conflicts. This is why the current crisis, no matter what its outcome may be, has already created a new international situation. (Emphasis added.)”

That first sentence is utterly blind to U.S.—and possibly British—actions in the last half-century. It’s utterly flabbergasting. How could anyone with a passing knowledge of history say that we have been “spared military conflicts”? Oh, I suppose he means that war no longer takes place on western soil. That it is waged by western powers seemingly everywhere else doesn’t enter into it. And the subtlety of economic warfare is something not even worth raising, if this guy can’t even see the bullets and bombs that are literally everywhere.

Instead, this fool heaps his entire opprobrium on Putin’s shoulders, putting a halo on the rest of the world and white-washing history to a degree that makes me think he’s suffered a serious head injury.

From that sentence alone, there is no need to take this guy seriously. A pity he put it all the way at the end of the article or I could have saved myself some time, I suppose.

1 week Ago

Schema migration in Quino 1.13

Published by marco on in Programming

Quino is a metadata framework for .NET. It provides a means of defining an application-domain model in the form of metadata objects. Quino also provides many components and support libraries that work with that metadata to automate many services and functions. A few examples are an ORM, schema migration, automatically generated user interfaces and reporting tools.

The schema-migration tool

The component we’re going to discuss is the automated schema-migration for databases. A question that recently came up with a customer was: what do all of the options mean in the console-based schema migrator?

Here’s the menu you’ll see in the console migrator:

Advanced Options
(1) Show migration plan
(2) Show significant mappings
(3) Show significant mappings with unique ids
(4) Show all mappings
(5) Show all mappings with unique ids

Main Options
(R) Refresh status
(M) Migrate database
(C) Cancel

The brief summary is:

  • The only action that actually makes changes is (M)
  • Option (1) is the only advanced option you will every likely use; use this to show the changes that were detected

The other advanced options are more for debugging the migration recommendation if something looks wrong. In order to understand what that means, we need to know what the migrator actually does.

 Schema migration overview

  1. Provide the application model as input
  2. Import a model from the database as input
  3. Generate a mapping between the two models
  4. Create a migration plan to update the database to reflect the application model
  5. Generate a list of commands that can be applied to the database to enact the plan
  6. Execute the commands against the database

The initial database-import and final command-generation parts of migration are very database-specific. The determination of differences is also partially database-specific (e.g. some databases do not allow certain features so there is no point in detecting a difference that cannot ever be repaired). The rest of the migration logic is database-independent.

Gathering data for migration

The migrator works with two models: the target model and a source model

  • The target model is provided as part of the application and is usually loaded from a core assembly. The source model is imported from the database schema by the “import handler”

Given these two models, the “mapping builder” creates a mapping. In the current implementation of Quino, there is no support for allowing the user to adjust mapping before a migration plan is built from it. However, it would be possible to allow the user to verify and possibly adjust the mapping. Experience has shown that this is not necessary. Anytime we thought we needed to adjust the mapping, the problem was instead that the target model had been configured incorrectly. That is, each time we had an unexpected mapping, it led us directly to a misconfiguration in the model.

The options to show mappings are used to debug exactly such situations. Before we talk about mapping, though, we should talk about what we mean by “unique ids”. Every schema-relevant bit of metadata in a Quino model is associated with a unique id, in the form of a Guid and called a “MetaId” in Quino.

Importing a model from a database

What happens during when the import handler generates a model?

The importer runs in two phases:

  1. Extract the “raw model” from the database schema
  2. Enhance the “raw model” with data pulled from the application-specific Quino metadata table in the same database

A Quino application named “demo” will have the following schema:

  • All modeled tables are named “demo__*”
  • The metadata table is named “demometadata__elementdescription”

The migrator reads the following information into a “raw model”

  • Tables => MetaClasses
  • Fields/Columns => MetaProperties
  • Indexes => MetaIndexes
  • Foreign Keys => MetaPaths

If there is no further information in the database, then the mapper will have to use the raw model only. If, however, the database was created or is being maintained by Quino, then there is additional information stored in the metadata table mentioned above. The importer enhanced the raw model with this information, in order to improve mapping and difference-recognition. The metadata table contains all of the Quino modeling information that is not reflected in a standard database schema (e.g. the aforementioned MetaId).

The data available in this table is currently:

  • SchemaIdentifier: the identifier used in the raw model/database schema
  • Identifier: the actual identifier of the metadata element that corresponds to the element identified by the SchemaIdentifier
  • MetaId: the unique id for the metadata element
  • ObjectType: the type of metadata (one of: class, property, index, path, model)
  • ParentMetaId: the unique id of the metadata element that is the logical parent of this one; only allowed to be empty for elements with ObjectType equal to “model”
  • Data: Custom data associated with the element, as key/value pairs
  • DataVersion: Identifies the format type of the “Data” element (1.0.0.0 corresponds to CSV)

For each schema element in the raw model, the importer does the following:

  1. Looks up the data associated with that SchemaIdentifier and ObjectType (e.g. “punchclock__person” and “class”)
  2. Updates the “Identifier”
  3. Sets the “MetaId”
  4. Loads the key/value pairs from the Data field and applies that data to the element

Generating a mapping

At this point, the imported model is ready and we can create a mapping between it and the application model. The imported model is called the source model while the application model is called the target model because we’re migrating the “source” to match the “target”.

We generate a mapping by iterating the target model:

  1. Find the corresponding schema element in the source model using MetaIds[1]
  2. If an element can be found, create a mapping for those two elements
  3. If no element can be found, create a mapping with the target element. This will cause the element to be created in the database.
  4. For all elements in the source model that have no corresponding element in the target model, create a mapping with only the source element. This will cause the element to be dropped from the database.

Creating a migration plan

The important decisions have already been made in the mapping phase. At this point, the migrator just generates a migration plan, which is a list of differences that must be addressed in order to update the database to match the target model.

  • If the mapping has a source and target element
    • Create a difference if the element has been renamed
    • Create a difference if the element has been altered (e.g. a property has a different type or is now nullable; an index has new properties or is no longer unique; etc.)
    If the mapping has only a source, generate a difference that the element is unneeded and should be dropped.
  • If the mapping has only a target, generate a difference that the element is missing and should be created.

This is the plan that is shown to the user by the various migration tools available with Quino.[2]

The advanced console-migrator commands

At this point, we can now understand what the advanced console-migrator commands mean. Significant mappings are those mappings which correspond to a difference in the database (create, drop, rename or alter).

  • Show significant mappings: show significant mappings to see more detail about the names on each side
  • Show significant mappings with unique ids: same as above, but also include the MetaIds for each side. Use this to debug when you suspect that you might have copy/pasted a MetaId incorrectly or inadvertently moved one.
  • Show all mappings: Same detail level as the first option, but with all mappings, including those that are 100% matches
  • Show all mappings with unique ids: same as above, but with MetaIds

As already stated, the advanced options are really there to help a developer see why the migrator might be suggesting a change that doesn’t correspond to expectations.

Generating commands for the plan

At this point, the migrator displays the list of differences that will be addressed by the migrator if the user chooses to proceed.

What happens when the user proceeds? The migrator generates database-specific commands that, when executed against the database, will modify the schema of the database.[3]

Commands are executed for different phases of the migration process. The phases are occasionally extended but currently comprise the following.

  • Initialize: perform any required initialization before doing anything to the schema
  • DropConstraintsAndIndexes: drop all affected constraints and indexes that would otherwise prevent the desired modification of the elements involved in the migration.
  • AddUpdateOrRenameSchema: Create new tables, columns and indexes and perform any necessary renaming. The changes in this phase are non-destructive
  • UpdateData: Perform any necessary data updates before any schema elements are removed. This is usually the phase in which custom application code is executed, to copy existing data from other tables and fields before they are dropped in the next phase. For example, if there is a new required 1–1 relation, the custom code might analyze the other data in the rows of that table to determine which value that row should have for the new foreign key.
  • DropSchema: Drop any unneeded schema elements and data
  • CreatePrimaryKeys: Create primary keys required by the schema. This includes both new primary keys as well as reestablishing primary keys that were temporarily dropped in the second phase.
  • CreateConstraintsAndIndexes: Create constraints and indexes required by the schema. This includes both new constraints and indexes as well as reestablishing constraints and indexes that were temporarily dropped in the second phase.
  • UpdateMetadata: Update the Quino-specific metadata table for the affected elements.

Executing the migration plan

The commands are then executed and the results logged.

Afterward, the schema is imported again, to verify that there are no differences between the target model and the database. In some (always rarer) cases, there will still be differences, in which case, you can execute the new migration plan to repair those differences as well.

In development, this works remarkably well and often, without further intervention.

Fixing failed migrations

In some cases, there is data in the database that, while compatible with the current database schema, is incompatible with the updated schema. This usually happens when a new property or constraint is introduced. For example, a new required property is added that does not have a default value or a new unique index is added which existing data violates.

In these cases, there are two things that can be done:

  • Either the database data is cleaned up in a way that makes it compatible with the target schema[4]
  • Or the developer must add custom logic to the metadata elements involved. This usually means that the developer must set a default value on a property. In rarer cases, the developer must attach logic to the affected metadata (e.g. the property or index that is causing the issue) that runs during schema migration to create new data or copy it from elsewhere in order to ensure that constraints are satisfied when they are reestablished at the end of the migration.

In general, it’s strongly advised to perform a migration against a replica of the true target database (e.g. a production database) in order to guarantee that all potential data situations have been anticipated with custom code, if necessary.

Quino Migration versus EF Migrations

It’s important to point out that Quino’s schema migration is considerably different from that employed by EF (which it picked up from the Active Migrations in Ruby, often used with Ruby on Rails). In those systems, the developer generates specific migrations to move from one model version to another. There is a clear notion of upgrading versus downgrading. Quino only recognizes migrating from an arbitrary model to another arbitrary model. This makes Quino’s migration exceedingly friendly when moving between development branches, unlike EF, whose deficiencies in this area have been documented.


[1] The default is to use only MetaIds. There is a mode in which identifiers are used as a fallback but it is used only for tools that import schemas that were not generated by Quino. Again, if the Quino metadata table hasn’t been damaged, this strict form of mapping will work extremely well.
[2] The Winform and Web user interfaces for Quino both include built-in feedback for interacting with the schema migration. There are also two standalone tools to migrate database schemas: a Winform application and a Windows console application.
[3] The form of these commands is currently a mix of SQL and custom C# code. A future feature of the migration will be to have all commands available as SQL text so that the commands, instead of being executed directly, could be saved as a file and reviewed and executed by DBAs instead of letting the tool do it. We’re not quite there yet, but proceeding nicely.
[4] This is generally what a developer does with his or her local database. The data contained therein can usually be more or less re-generated. If there is a conflict during migration, a developer can determine whether custom code is necessary or can sometimes determine that the data situation that causes the problem isn’t something that comes up in production anyway and just remove the offending elements or data until the schema migration succeeds.

2 weeks Ago

EF Migrations troubleshooting

Published by marco on in Programming

The version of EF Migrations discussed in this article is 5.0.20627. The version of Quino is less relevant: the features discussed have been supported for years. For those in a hurry, there is a tl;dr near the end of the article.

We use Microsoft Entity Framework (EF) Migrations in one of our projects where we are unable to use Quino. We were initially happy to be able to automate database-schema changes. After using it for a while, we have decidedly mixed feelings.

As developers of our own schema migration for the Quino ORM, we’re always on the lookout for new and better ideas to improve our own product. If we can’t use Quino, we try to optimize our development process in each project to cause as little pain as possible.

EF Migrations and branches

We ran into problems in integrating EF Migrations into a development process that uses feature branches. As long as a developer stays on a given branch, there are no problems and EF functions relatively smoothly.[1]

However, if a developer switches to a different branch—with different migrations—EF Migrations is decidedly less helpful. It is, in fact, quite cryptic and blocks progress until you figure out what’s going on.

Assume the following not-uncommon situation:

  • The project is created in the master branch
  • The project has an initial migration BASE
  • Developers A and B migrate their databases to BASE
  • Developer A starts branch feature/A and includes migration A in her database
  • Developer B starts branch feature/B and includes migration B in his database

We now have the situation in which two branches have different code and each has its own database schema. Switching from one branch to another with Git quickly and easily addresses the code differences. The database is, unfortunately, a different story.

Let’s assume that developer A switches to branch feature/B to continue working there. The natural thing for A to do is to call “update-database” from the Package Manager Console[2]. This yields the following message—all-too-familiar to EF Migrations developers.

 EF Migrations pending-changes warning

“Unable to update database to match the current model because there are pending changes and automatic migration is disabled. Either write the pending changes to a code-based migration or enable automatic migration. […]”

This situation happens regularly when working with multiple branches. It’s even possible to screw up a commit within a single branch, as illustrated in the following real-world example.

  • Add two fields to an existing class
  • Generate a migration with code that adds two fields
  • Migrate the database
  • Realize that you don’t need one of the two fields
  • Remove the C# code from the migration for that field
  • Tests run green
  • Commit everything and push it

As far as you’re concerned, you committed a single field to the model. When your co-worker runs that migration, it will be applied, but EF Migrations immediately thereafter complains that there are pending model changes to make. How can that be?

Out-of-sync migrations != outdated database

Just to focus, we’re actually trying to get real work done, not necessarily debug EF Migrations. We want to answer the following questions:

  1. Why is EF Migrations having a problem updating the schema?
  2. How do I quickly and reliably update my database to use the current schema if EF Migrations refuses to do it?

The underlying reason why EF Migrations has problems is that it does not actually know what the schema of the database is. It doesn’t read the schema from the database itself, but relies instead on a copy of the EF model that it stored in the database when it last performed a successful migration.

That copy of the model is also stored in the resource file generated for the migration. EF Migrations does this so that the migration includes information about which changes it needs to apply and about the model to which the change can be applied.

If the model stored in the database does not match the model stored with the migration that you’re trying to apply, EF Migrations will not update the database. This is probably for the best, but leads us to the second question above: what do we have to do to get the database updated?

Generate a migration for those “pending changes”

The answer has already been hinted at above: we need to fix the model stored in the database for the last migration.

Let’s take a look at the situation above in which your colleague downloaded what you thought was a clean commit.

From the Package Manager Console, run add-migration foo to scaffold a migration for the so-called “pending changes” that EF Migrations detected. That’s interesting: EF Migrations thinks that your colleague should generate a migration to drop the column that you’d only temporarily added but never checked in.

That is, the column isn’t in his database, it’s not in your database, but EF Migrations is convinced that it was once in the model and must be dropped.

How does EF Migrations even know about a column that you added to your own database but that you removed from the code before committing? What dark magic is this?

The answer is probably obvious: you did check in the change. The part that you can easily remove (the C# code) is only half of the migration. As mentioned above, the other part is a binary chunk stored in the resource file associated with each migration. These BLOBS are stored in the table _MigrationHistory table in the database.

 DB Migration History table Migration data is stored as binary data

How to fix this problem and get back to work

Here’s the tl;dr: generate a “fake” migration, remove all of the C# code that would apply changes to the database (shown below) and execute update-database from the Package Manager Console.

 An empty migration

This may look like it does exactly nothing. What actually happens is that it includes the current state of the EF model in the binary data for the last migration applied to the database (because you just applied it).

Once you’ve applied the migration, delete the files and remove them from the project. This migration was only generated to fix your local database; do not commit it.

Everything’s cool now, right?

Applying the fix above doesn’t mean that you won’t get database errors. If your database schema does not actually match the application model, EF will crash when it assumes fields or tables are available which do not exist in your database.

Sometimes, the only way to really clean up a damaged database—especially if you don’t have the code for the migrations that were applied there[3]—is to remove the misapplied migrations from your database, undo all of the changes to the schema (manually, of course) and then generate a new migration that starts from a known good schema.

Conclusions and comparison to Quino

The obvious answer to the complaint “it hurts when I do this” is “stop doing that”. We would dearly love to avoid these EF Migrations-related issues but developing without any schema-migration support is even more unthinkable.

We’d have to create upgrade scripts manually or would have to maintain scripts to generate a working development database and this in each branch. When branches are merged, the database-upgrade scripts have to be merged and tested as well. This would be a significant addition to our development process, has maintainability and quality issues and would probably slow us down even more.

And we’re certainly not going to stop developing with branches, either.

We were hoping to avoid all of this pain by using EF Migrations. That EF Migrations makes us think of going back to manual schema migration is proof that it’s not nearly as elegant a solution as our own Quino schema migration, which never gave us these problems.

Quino actually reads the schema in the database and compares that model directly against the current application model. The schema migrator generates a custom list of differences that map from the current schema to the desired schema and applies them. There is user intervention but it’s hardly ever really required. This is an absolute godsend during development where we can freely switch between branches without any hassle.[4]

Quino doesn’t recognize “upgrade” versus “downgrade” but instead applies “changes”. This paradigm has proven to be a much better fit for our agile, multi-branch style of development and lets us focus on our actual work rather than fighting with tools and libraries.


[1] EF Migrations as we use it is tightly bound to SQL Server. Just as one example, the inability of SQL Server to resolve cyclic cascade dependencies is in no way shielded by EF Migrations. Though the drawback originates in SQL Server, EF Migrations simply propagates it to the developer, even though it purports to provide an abstraction layer. Quino, on the other hand, does the heavy lifting of managing triggers to circumvent this limitation.
[2] As an aside, this is a spectacularly misleading name for a program feature. It should just be called “Console”.
[3] I haven’t ever been able to use the Downgrade method that is generated with each migration, but perhaps someone with more experience could explain how to properly apply such a thing. If that doesn’t work, the method outlined above is your only fallback.
[4] The aforementioned database-script maintenance or having only very discrete schema-update points or maintaining a database per branch and switching with configuration files or using database backups or any other schemes that end up distracting you from working.

3 weeks Ago

When you really start learning

Published by marco on in Quotes

“The paradox of education is precisely this—that as one begins to become conscious, one begins to examine the society in which he is being educated.”