A chapter on Modules in JavaScript 6 from Exploring JS book - link
Wednesday, July 20, 2016
Saturday, July 16, 2016
CloudRail and Crashlytics
In mobile apps, adding one thing leads to another and so I ended up using Twitter's Fabric / Crashlytics.
Prior to that, I was lucky to have the good guys from CloudRail introduce their fantastic library that communicates with four major cloud storage providers (Dropbox, Google Drive, OneDrive, and Box). This came about at the same time Dropbox announced the retirement of v1 of their APIs. As the most requested features for MoneyManagerEx for Android were the support for some of these services, this was a great match. A unified interface to different cloud providers was something I was expecting to create in order to provide multiple storage providers for data synchronization anyway, and architecturally this solution fit perfectly into the app's roadmap.
After adding this, of course, various issues appeared. This is almost a given on even the quietest releases. Just after Beta testing for at least a week does not produce any exceptions, rest assured that publishing an app to Production will result in a torrent of exception logs. Various devices, operating systems, and network environments, mixed with different user levels will invariably result in infinite number of possibilities for exceptions.
This torrent of exception reports is becoming an overkill for a custom default uncaught exception handler. Plus the custom solution depends heavily on users actually willing to submit the report via email. Over the past several months, this solution really shows its age and is becoming unmaintainable as the number of users constantly grows. And, as I've found out only later, the stack trace produced in this manner does not include all the useful information. With multithreading being used ever more for even the simplest tasks, the log covering the crashing thread does not produce the full picture of what exactly went wrong, especially if there are several layers of libraries. So, an issue with SSL parsing in OpenSSL might cause OneDrive part of CloudRail to fail, in turn resulting in an exception during the initialization of the sync service which is set up to use Dropbox as the provider!
This level of logging only becomes possible by introducing a specialized automated exception reporting solution.
This lead to dusting off my Fabric account and adding Crashlytics to the app. The results are simply stunning. Real-time reports show the exceptions as they happen. Now, without relying on users' submission, I do not need to bug users to send exception reports manually any longer.
Another great benefit is that the caught exceptions can now be reported, too. Adding a whole bunch of try/catch blocks muted these errors and prevented the app from crashing but in reality did nothing to handle the root cause. Now I can log any of these caught exceptions and report them. Then, whichever one tops the list, it will get addressed and the related code improved further. Specific cases, like missing files, DNS errors, and so on, can be specifically handled and the user notified. This way they can be properly handled and excluded from the exception logs.
So, overall, lots of fun introducing two new features/libraries - CloudRail and Crashlytics. It is one of those moments when one asks himself how the world existed before this. Should I mention that both are free to use, even though they are not Open Source? If they remain so, they will probably have a historic impact on technology implementation by improving outreach (CloudRail) and handling the user experience quality (Crashlytics). Thanks, guys!
Happy development to you all.
Prior to that, I was lucky to have the good guys from CloudRail introduce their fantastic library that communicates with four major cloud storage providers (Dropbox, Google Drive, OneDrive, and Box). This came about at the same time Dropbox announced the retirement of v1 of their APIs. As the most requested features for MoneyManagerEx for Android were the support for some of these services, this was a great match. A unified interface to different cloud providers was something I was expecting to create in order to provide multiple storage providers for data synchronization anyway, and architecturally this solution fit perfectly into the app's roadmap.
After adding this, of course, various issues appeared. This is almost a given on even the quietest releases. Just after Beta testing for at least a week does not produce any exceptions, rest assured that publishing an app to Production will result in a torrent of exception logs. Various devices, operating systems, and network environments, mixed with different user levels will invariably result in infinite number of possibilities for exceptions.
This torrent of exception reports is becoming an overkill for a custom default uncaught exception handler. Plus the custom solution depends heavily on users actually willing to submit the report via email. Over the past several months, this solution really shows its age and is becoming unmaintainable as the number of users constantly grows. And, as I've found out only later, the stack trace produced in this manner does not include all the useful information. With multithreading being used ever more for even the simplest tasks, the log covering the crashing thread does not produce the full picture of what exactly went wrong, especially if there are several layers of libraries. So, an issue with SSL parsing in OpenSSL might cause OneDrive part of CloudRail to fail, in turn resulting in an exception during the initialization of the sync service which is set up to use Dropbox as the provider!
This level of logging only becomes possible by introducing a specialized automated exception reporting solution.
This lead to dusting off my Fabric account and adding Crashlytics to the app. The results are simply stunning. Real-time reports show the exceptions as they happen. Now, without relying on users' submission, I do not need to bug users to send exception reports manually any longer.
Another great benefit is that the caught exceptions can now be reported, too. Adding a whole bunch of try/catch blocks muted these errors and prevented the app from crashing but in reality did nothing to handle the root cause. Now I can log any of these caught exceptions and report them. Then, whichever one tops the list, it will get addressed and the related code improved further. Specific cases, like missing files, DNS errors, and so on, can be specifically handled and the user notified. This way they can be properly handled and excluded from the exception logs.
So, overall, lots of fun introducing two new features/libraries - CloudRail and Crashlytics. It is one of those moments when one asks himself how the world existed before this. Should I mention that both are free to use, even though they are not Open Source? If they remain so, they will probably have a historic impact on technology implementation by improving outreach (CloudRail) and handling the user experience quality (Crashlytics). Thanks, guys!
Happy development to you all.
Monday, July 4, 2016
CloudRail: Unified Cloud Storage API
Just came across CloudRail library for Android and am adding it to MoneyManagerEx for Android. The library provides a unified API to several cloud storage (and other) services.
After Dropbox announced deprecation of v1 of their APIs and libraries, these needed to be replaced in the app. Now, for the same amount of effort in migration the app will get, for free, the ability to use four of the most common cloud storage providers. This is fantastic! Hope this stays a secret. :)
https://cloudrail.com/
After Dropbox announced deprecation of v1 of their APIs and libraries, these needed to be replaced in the app. Now, for the same amount of effort in migration the app will get, for free, the ability to use four of the most common cloud storage providers. This is fantastic! Hope this stays a secret. :)
https://cloudrail.com/
Wednesday, June 29, 2016
Monday, June 27, 2016
Deploy SQL Migrations with Octopus
Here is a brief summary and a to-do list for packing and deploying SQL migrations in combination with Octopus. The prerequisites are: .NET project, Fluent Migrator, Octopus Deploy.
The first step is to set-up database migrations using code (C#). For this,
- create a library project in the solution,
- add Fluent Migrator (eventually FluentMigrator.Tools and/or FluentMigrator.Runner but these are not required) NuGet package to it,
- create the first migration class containing the change (schema and/or data).
This will produce the migration binary .dll file on build operation. To get the SQL commands for the actual change, one needs to run the Migrator tool to generate the change commands based on the current version of the database. If this is the first migration it will also generate the commands for creation of the Migration_Info table.
- Create SQL generation script(s). There are a few useful scripts for checking the SQL output.
- Script for previewing the current migration (just runs -preview),
- Script for manually generating the SQL output, and
- PowerShell script for automatic generation of the SQL file output during the automatic build process.
The PowerShell script will be executed after the build completes on an integration server. It will generate the SQL file from the built migrations .dll binary. To do that,
- add the post-build script that runs the Migrator and generates the output SQL file (this is the automatic script from above).
After this, it is time to generate the NuGet package.
- create .nuspec file that includes the generated .sql file.
- adjust this script to create the NuGet package with the generated output.
- set the script to upload the NuGet package to a NuGet repository. This is the Octopus Library.
Once this is done, the only remaining step is the deployment.
- create the Deploy.ps1 script and include it in the generated NuGet package. This script should either execute the SQL file or copy/send it to the DBAs for examination and scheduled execution.
This solution provides the history of database changes (both schema and data). It is automatic and does not require developers to manipulate or send SQL commands. Besides the code migrations, it also creates an archive of all the migration packages, which is useful for the database administrators and possibly other parties. These packages can be safely stored along with the other binary packages for deployment.
The first step is to set-up database migrations using code (C#). For this,
- create a library project in the solution,
- add Fluent Migrator (eventually FluentMigrator.Tools and/or FluentMigrator.Runner but these are not required) NuGet package to it,
- create the first migration class containing the change (schema and/or data).
This will produce the migration binary .dll file on build operation. To get the SQL commands for the actual change, one needs to run the Migrator tool to generate the change commands based on the current version of the database. If this is the first migration it will also generate the commands for creation of the Migration_Info table.
- Create SQL generation script(s). There are a few useful scripts for checking the SQL output.
- Script for previewing the current migration (just runs -preview),
- Script for manually generating the SQL output, and
- PowerShell script for automatic generation of the SQL file output during the automatic build process.
The PowerShell script will be executed after the build completes on an integration server. It will generate the SQL file from the built migrations .dll binary. To do that,
- add the post-build script that runs the Migrator and generates the output SQL file (this is the automatic script from above).
After this, it is time to generate the NuGet package.
- create .nuspec file that includes the generated .sql file.
- adjust this script to create the NuGet package with the generated output.
- set the script to upload the NuGet package to a NuGet repository. This is the Octopus Library.
Once this is done, the only remaining step is the deployment.
- create the Deploy.ps1 script and include it in the generated NuGet package. This script should either execute the SQL file or copy/send it to the DBAs for examination and scheduled execution.
This solution provides the history of database changes (both schema and data). It is automatic and does not require developers to manipulate or send SQL commands. Besides the code migrations, it also creates an archive of all the migration packages, which is useful for the database administrators and possibly other parties. These packages can be safely stored along with the other binary packages for deployment.
Thursday, June 23, 2016
Process Hacker
Process Hacker is an Open Source version of Process Explorer. Some say it has more functionality, too. To be tested.
Project page
Project page
Friday, June 17, 2016
Subscribe to:
Posts (Atom)