Uploading a large amount of small files to S3 quickly
TL;DR
Make sure you have Python 3 installed and run:
# install pip install boto pip install python-magic brew install libmagic // this one got me for a second as I'm running an M1 Mac # get the script and allow it to be executed curl https://gist.githubusercontent.com/darbio/8429cb82c00b7d4a2d9069a10f9f08bd/raw/28da5aba3935b05bde3e6d6b96b110b5e8bd3d2b/s3-parallel-put.py --output s3-parallel-put.py chmod +x ./s3-parallel-put.py # execute ./s3-parallel-put.py --bucket=BUCKET_NAME ./source-directory The detail
I recently had a solution which required the generation...
SwiftUI binding in UIKit with UIHostingController and ObservableObject
Often I have found myself wanting to use SwiftUI data bindings in Views hosted via UIHostingController in UIKit. A great example of this is when you want to use a SwiftUI View in a MKAnnotationView which has a data binding in the SwiftUI View.
Obviously, UIKit and SwiftUI don’t work with the @State of @Binding property wrappers, however @ObservedObject still works well and changes made on the UIKit code...
ThreadPool exceptions in IIS Express
When debugging a WebAPI application recently which used an asynchronous library, I came accross a situation where the application kept on throwing the following InvalidOperationException:
There were not enough free threads in the ThreadPool to complete the operation.
Running the following code from a WebAPI controller:
Results in the following:
workers = 2
completions = 2
This is too low to run any other async...
Event-based architecture using AWS (SNS and SQS) and C#
Introduction
The following code is an example of an event based architecture using an SNS topic per event. It consists of:
Producer- A system which produces events to an SNS topic.Consumer- A system which consumes messages placed onto an SQS queue by an SNS topic.
The two events we are interested in are:
INCIDENT_CREATEDINCIDENT_UPDATED
We will create consumers...
Lookup lists (using MongoDB)
Darb.io Lookups (in MongoDB)
An example of how to do lookups in MongoDB, without persisting the data or having to maintain relationships etc. in your services.
Background
In an object model we often want to specify a domain type (as in ‘domain classification’, not object type) to an object.
Example
Take the example of an Organization. In our domain, Organizations have a name, and an organization type (classification). The organization...
WebAPI and ADFS as external login provider
Setting up ADFS
For these steps, I presume that you are running ADFS 3 on Windows Server 2012 R2 and that it has been already installed and configured.
Setting up the Relying Party Trust
- Open ADFS Managememt
- Expand ‘Trust Relationships’ and click on ‘Relying Party Trusts’.
- Click on ‘Add Relying Party Trust’ in the right hand panel.
- Click ‘Start’ and...
API best practice design
Here follows an aggregation of some generic rules for generating a Web API to program against. Based upon a number of references, supplied at the bottom of this post.
API design rules
Use REST verbs
Use HTTP verbs which make sense from the perspective of the API consumer:
GETwill retrieve an resourcePOSTwill create a resourcePUTwill update a resourcePATCH...
Architecting on AWS - Day 3, Sydney
These are my notes from day 3 of the Amazon ‘Architecting on AWS’ course which was run in Sydney in March 2014. The trainer/presenter was John Rotenstein from AWS.
Simple Queue Service
- Regionally based
- Automatically replicated throughout AZs
- Concept of dead letter queues
- Eventually consistent
- FIFO, kind of…
- “Visibility Timeout” is the invisibility timeout
- Can be overridden in code
- Messages can be retained for up to 14 days
- Messages are...
Architecting on AWS - Day 2, Sydney
These are my notes from day 2 of the Amazon ‘Architecting on AWS’ course which was run in Sydney in March 2014. The trainer/presenter was John Rotenstein from AWS.
Route 53
- AWS DNS provider
- Split DNS? Is this possible?
- Subdomains can CNAME to an aws address (e.g. assets.s3.aws.com)
Routing
- Round robin
- Weighted
- 0 will stop traffic going to that address
- Equal
- Weighted
- Latency based
Architecting on AWS - Day 1, Sydney
These are my notes from day 1 of the Amazon ‘Architecting on AWS’ course which was run in Sydney in March 2014. The trainer/presenter was John Rotenstein from AWS.
Best Practices
Design for failure
- Design for failure, nothing fails. E.g. Aviation, seat belts, “black box”
- “Everything fails, all the time”
- Avoid single points of failure
Loose coupling sets you free
- SOA
- More loosely they couple, the bigger they scale ...
XmlSerialization considerations (on MonoTouch)
I recently was making an app which needed to store data on the client. The obvious choices were:
- Local file storage (e.g. serialise a class to
XMLand store it inDocuments) - Use an
SQLitedatabase to store the data and read/write when needed
I have done both before, and generally find that the XML storage is a quick and easy method of storing data...
FDI Calculator App in iOS AppStore
My first (personally funded and created) app is now up in the Apple App Store.
The app is a tool for:
- Fire Danger Index calculator
- Fire Danger Rating information
- Drought Factor calculator
From the AppStore app description:
FDI Utility is an application to calculate the Forest Fire Danger Index (FFDI), Grass Fire Danger Index (GFDI) and Drought Index (KBDI). It displays the results as the Fire Danger Index (FDI) and also...
Run tests from networked location
I use Parallels on OSX to run Visual Studio inside a VM.
With a test project I had written, I was getting the following exception thrown
Unit Test Adapter threw exception: URI formats are not supported.. Which was highly annoying!
After some googling, I found the solution:
- Double click on Local.testsettings which is under the Solution Items of Project
- Test Settings window is displayed. Click on the...
Compile mono from source on OSX
The following is how I compiled mono latest from source:
- Install MacPorts from http://macports.org
- Install gettext
sudo port install gettext
- Locate your current mono environment. It is usually at:
/Library/Framework/Mono.framework/Versions/Current
- Download the mono source from GitHub master branch
git clone https://github.com/mono/mono.git
- Run autogen with your OSX version prefix
./autogen.sh --prefix=/Library/Framework/Mono.framework/Versions/Current
- Run make
sudo make
- Optionally, you can run the tests
sudo make check
- Run make install
sudo...
Mercurial Branching
We are currently going down the path of switching from our huge monolithic SVN repository, to a number of product based Mercurial repositories. Historically, I have used FogBugz and Kiln, but this time we have decided to give the JIRA stack a whirl - mainly for GrassHopper, as one of the project managers here wanted to give it a go.
In the old SVN repo, instead of branching and tagging, we copied files to “Release...
Microsoft Azure DevCamp in Sydney
Some may know that I went along to the Azure DevCamp in Sydney yesterday.
I have some plans for Azure, and hadn’t really had a chance to have a play with it before yesterday… Well.. I was impressed, and had a lot of fun!
The new portal is pretty schmick! And makes it easy to create:
- Websites
- Cloud Services
- Storage
- SQL DB's
Best thing is that, during preview, it is free for 90...
Modal loading dialog in MonoTouch
Update: I was using this in a project which called the UILoadingView class from an anonymous delegate and kept coming up against the dreaded SIGSEGV error… Well it was a silly mistake on my behalf. I forgot to wrap the DismissWithClickedButtonIndex(0, true); in BeginInvokeOnMain (...). Code updated below.
As some of you may know, I work as a Solutions Architect at IQX in Sydney - a company specialising in SAP integration with,...
Resources in MonoTouch assembly
In my experience, I found that adding resources into a MonoTouch project can be slightly confusing - if you have not done it before.
I had a requirement to embed images into a project… easy you may say. Add a solution folder into the solution, add your images into the folder and give them a build action of ‘Content’. Duh… Why bother with a blog post? I hear the crowd mutter.
Well… It on the...
jQuery Validate and MaxLength
A quick one… Just had a situation where we were adding the maxlength attribute to a html input, and then trying to use the excellent jQuery Validate plugin to validate the form… Resulting in an invalid form! Doh!!
Seems it is a bug…
Doh! Hopefully fixed soon.
Happy coding!!
Handling .Net DateTime in jQueryUI DatePicker using KnockoutJS
In a recent project we were using KnockoutJS to bind a .Net model to a JavaScript ViewModel to a HTML View.
The View has a number of jQuery UI controls on it, including the infamous the DatePicker.
The model had a number of ‘complex’ types on it, however the type which gave us the most headache was the ‘simple’ .Net CLR DateTime type.
When the JavaScriptSerializer serializes a .Net...
Sending an email with an attachment from Workflow Foundation
The problem:
The SendEmailActivity in WF sucks… because you can’t send an email with attachments from it.
I want to be able to send an email from a SharePoint WorkFlow, using the SP Outbound Mail Service and adding some attachments to the message.
The workaround:
Use the .Net MailMessage class (System.Net.Mail) and a Code activity to send the email:
Non-SharePoint example:
SharePoint example:
...
500 Internal Server Errors when using jQuery.ajax in MOSS SharePoint 2007 SP1
I have recently been architecting a SharePoint forms solution which uses jQuery.ajax To communicate, using JSON, with a .Net 2.0 web service - hosted in the \_vti\_bin / ISAPI folder of a SharePoint host.
For various reasons we have been developing in SharePoint 2010, and it has worked fine in our dev environments. Today we did our first preliminary deployment of the solution into the client’s development environment…
After some fun and games...