4th Annual AMQP Conference Announced

August 30, 2011 – 7:16 am

The AMQP Working Group has been working diligently for over 5 years and we have released several versions of the protocol, including 0.8, 0.91 and more recently the 0.10 release.

StormMQ joined the working Group in May 2010 around the time that the AMQP 1.0 specification was starting to firm up. After all this work we are now ready to release the 1.0 specification as ‘Final’.

To help celebrate this milestone the Working Group announced the date of their Fourth Annual Conference and AMQP 1.0 Launch Event. This will take place on 12th October 2011 at the offices of JPMC on Park Avenue, New York.

So, who should attend? This event is focused at IT directors of large organisations such as Banks, Trading Houses, Stock Exchanges, Manufacturers, Government agencies and the IT Directors of companies that supply, support and manage IT infrastructure for these types companies.

So, why should you attend? Here is a short list of the things that will take place at the event:

  • Presentations by JPMorgan Chase, Bank of America Merrill Lynch, and Deutsche Börse.
  • Hearing directly from AMQP vendors about their technologies, and from end users on how they are benefiting from them.
  • Interactive panel session with vendors, users and analysts to learn the answers you need.
  • Technical session on AMQP 1.0 core protocol features.
  • AMQP 1.0 interoperability demonstration showcasing technology vendors including Red Hat Inc., VMware Inc., INETCO Systems Limited, Microsoft Corporation and StormMQ Ltd.

Tickets will be available soon, so if you would like to attend please get in contact.

Share

Whitepaper: Using AMQP based messaging to manage a cloud

August 25, 2011 – 7:07 pm
Anybody who has had to manage a large corporate IT network will know how hard it is to keep on top of monitoring. Here at StormMQ we had a rather large problem designing a monitoring system that could cope with our diverse infrastructure. This is because our network is cloud based with many nodes, based in many locations with many diverse functions.
Running a cloud requires a lot of hardware, software and man hours: servers, operating systems and administrators. Running a cloud profitably requires that those resources are used to maximum effect; so that not a CPU cycle is wasted, not one piece of hardware sits in storage too long and any one administrator can manage as much hardware as possible. In essence, a cloud needs to run itself with minimal intervention. Adding hardware, changing software and preventative maintenance must be automated, self-healing and self-updating: the nirvana of zero configuration!
Normal network monitoring tools such as Nagios and Cacti are not designed for a cloud based network where resources appear and disappear frequently. It is because of this that we had to design our own system, built internally using a AMQP messaging server. While designing this system we wrote a white paper. If you would like a copy you can download this here:
Using AMQP based messaging to manage a cloud
Share

AMQP 1.0 specification tracker

August 10, 2011 – 3:47 pm

The AMQP 1.0 protocol is almost ready to be passed back to the PMC for voting. Over the past few months this specificication has changed significantly and it is now changing on a weekly basis. Keeping track of all the activity is very hard….to help with this we have built an automated system ontop of Jenkins. We use this hevily internally, please feel free to do so too.

  • You can find a list of changes to the spec  HERE
  • The most recent version of the PDF document: HERE
Share

Some recent questions from customers and partners (part 2)

August 9, 2011 – 3:02 pm

We often get asked interesting questions from customers and partners. I try to keep a note of these and some of these I have already posted on this blog, however here are a few more.

Why put your MQ in the cloud?

Well why not? Seriously, think about it. Message queues allow you to move from the SQL -> Database -> File System model and into the network layer by affording:

  • Logical decoupling
  • Physical decoupling
  • Temporal decoupling

If you already have a distributed system, located in several locations, then why is jumping to the cloud so hard?

Why AMQP?

AMQP is not the only MQ protocol, it actually has a very small portion of the market, however, it is the most progressive and with the new 1.0 version of the protocol it is poised to grow significantly. This is because AMQP is an open, interoperable and reliable standard.

StormMQ is not the only supplier of AMQP services. AMQP is being used in many high performance clustering and grid computing projects all around the world including:

  • The Deutsche Börse
    “EUREX 12 is FIXML over AMQP Eurex is the very first exchange to introduce AMQP as a standard protocol on its system, thus easing the monitoring of positions and risk related data for its members and ensuring market integrity for all participants.”
  • JPMorgan
    Sends 1 billion AMQP messages per day; used in dozens of mission critical systems worldwide.
  • National Science Foundation
    The Ocean Observatories Initiative infrastructure uses sensor nets with AMQP to bring readings ashore from ocean platforms and a global pub-sub network to disseminate readings.
  • NASA
    The control plane of the Nebula Cloud Computing.
  • Mozilla
    Mozilla use RabbitMQ in Pulse, their in-house eventing and pub-sub bus.
  • INETCO
    Uses AMQP for passing real time data in cloud environments, between different components of the INETCO Insight product.
  • Smith Electric Vehicles
    Uses StormMQ’s AMQP cloud based service to monitor over 27,500 items of data per second on the worlds largest electric vehicle fleets.

Projects like this are massive, ground breaking and mission critical…and all using AMQP.

“AMQP is a good fit for cloud MaaS and other wide-area, Internet-based messaging purposes, because it defines the wire protocol and communication semantics without restricting the details of the messaging system implementation or the API. AMQP’s goal is to enable interoperability among disparate vendors’ platforms.”
- Roy Shulte (Gartner), Review of AMQP and other protocol options.

How interoperable are the different AMQP vendors?

I often get asked and challenged on the claims that AMQP is a truly interoperable protocol.

We can provide guarantees that will allow you to switch to another AMQP provider should you wish to do so – so confident are we of being interoperable. The whole idea and raison d’etre of AMQP is to give clients choice, choice means competition, means lower prices.

Share

The Magic Triangle; the future of AMQP

August 4, 2011 – 2:38 pm

As I have mentioned many times before, we have been working really hard on agreeing the specification and all of the related negotiations regarding the release of the AMQP 1.0 protocol. This is something that we are really passionate about for many reasons, not least because we plan to release an AMQP 1.0 compatible version of StormMQ along with a C Client called libamqp.

Earlier this month I took some time out and had a chat with John O’Hara, the founder of the AMQP Working Group and the Global Head of Architecture for Commodities at Bank of America Merrill Lynch. We covered many topics; however the overriding theme was regarding the future of AMQP and the MQ market in general.

We both agreed that the cloud based MQ market will most likely follow the trajectory of the web hosting market. 20 years ago you would have had to buy a server and install IIS, Zeus or some other expensive application to host your website. Doing so gave you a hugely configurable system that was often prohibitive in cost. Since then the market has commoditised significantly and now you can signup online and get a rather feature rich service for a few dollars a month. While cheap web hosting does not work for everybody, its availability has increased demand for websites, lowered prices and grown an industry of ‘mom and pop’ hosting companies….many of which have been swallowed up in market consolidation.

But, before the cloud based MQ market can take off we need a more broad adoption of AMQP by the general developer community. This needs native AMQP clients installed by default and widely supported by OS and hardware developers, put simply, you would need the full participation of the main operating system stack suppliers such as Microsoft, Redhat and Vmware. John described these companies as forming a ‘magic triangle’ of influence and luckily enough, each of these companies is committed to releasing an AMQP 1.0 compatible service or product. In doing so they will most probably release native clients and thus paving the way for broad adoption. It is the growth of pervasive AMQP clients and a strong and healthy developer community that companies like StormMQ will rely upon.

So, why should companies such as StormMQ, Inetco, Microsoft, Redhat & Vmware develop AMQP based services and products? Put simply, where is the money? We believe that customers will need some of the following services/products:

  • Stand alone brokers: Think of RabbitMQ and Apache QPid here…
  • Messaging Appliances: Wrapping it in sheet steel is a proven market, just as Solace / Cisco / EMC / NetApp
  • Management utilities to bring it all together; the Candle model
  • Proxies that allow people get off WebSphere and TIBCO products without burning their bridges; the Progress model.
  • Business transaction tracing solutions (the INETCO model).
  • Bastion technology – routers that bridge middleware between two firms that want to share information.
  • Cloud based services; StormMQ are the only supplier in this market at the moment, however there are two other suppliers getting ready to release services in this market.
  • Embedded devices: think of the ‘internet of things’ and a market around having a really cut down AMQP client printed onto silicone that can be used in tiny sensors and telemetry devices.

There are many more potential use cases for the AMQP protocol, but getting broad adoption of the protocol relies upon the Magic Triangle. We are in the early days of the commoditisation of AMQP, and as history has shown us, the only way is up from here.

Share

Security and privacy in the Cloud, how paranoid should Banks be?

May 25, 2011 – 6:53 am

The answer is very. Client data is always vitally important and combining that with financial details provides a red hot issue that Banks, not surprisingly take very, very seriously.

So when a supplier like StormMQ comes along and offers a Cloud based Message Queue Service, a discerning CIO will firstly take a view of the commercial proposition (which needs to be convincing) before even considering whether the Service will stand up to the scrutiny of his or her internal audit security teams.

Assuming the commercial proposition is attractive, the next stage is usually about where will my data be held and who has access? At this point a quality Cloud based provider should start to get excited (as we do at StormMQ) as we know to what lengths we have taken to build from the ground up a Service that offers 100% guarantee of data security that will convince even the most sceptical CIO out there……

For example:

Security and data encryption
We made the decision that our Services have to be 100% secure by default so that it is actually impossible to execute insecure ways of doing things. For example, you can’t use our Services without using SSL (and as things change, we’ll be locking down the choice of ciphers and secure hashes used, too). Internally, we only use IPSec – with IKE v2 the only choice – for all local network traffic. We use whole disk encryption, too.

End Point security
StormMQ Services can only be accessed over encrypted communications. Our webservice, REST API and AMQP end points all use 2048-bit SSL and support TLSv1. For our dedicated clusters we also offer private end points and an IPSec VPN and higher encryption strengths (if your operating system supports them). We have a number of controls at our entry points to identify and terminate disruptive traffic (‘DoS’ protection).

Network Security
All messages and out of band traffic through our Messaging Cloud uses mesh IPSec VPN tunnels with x.509 authentication and 256-bit keys.

Message Encryption
All messages, meta-information and AMQP ‘frames’ arriving at an end point are transmitted encrypted throughout our Messaging Cloud. All messages persisted for later delivery are encrypted on disk using AES-256 bit keys.

User Data
Critical account data encrypted in memory and is only encrypted-on-the-fly.

REST API Authentication
To guard against various REST attacks, all operations are invoked using HMAC-SHA256 signatures.

Passwords
Our system generates all passwords and secret keys. Hashing and Message Authentication algorithms do not rely on the partly compromised MD5 or SHA1 implementations.

Server Protection
Our servers are hardened, locked down and automated to become toasters in the event of compromise using best-of-breed practices. Back-end servers are web-inaccessible. Your IT Audit teams are welcome to review our server hardening.

All critical operational information is stored encrypted on disk using AES-256 keys.

Logging
All activity is logged. We provide all our clients with a full log of all their activity through their website portal.

And Finally….
The ability to locate and secure your data means so much to us that when you take up our Service, we will sign over ownership of the encrypted hard disks we use to you as an extra option. We provide a certificate of locality and ownership of data. At the end of your subscription, we will present the disks to you for secure destruction.

Access
We only allow a subset of SASL mechanisms, but, more importantly, enforce our password policy on our users. That way, we can ensure passwords are as secure as possible. The automated systems that use messaging don’t need memorable passwords for admin! We haven’t seen clients use LDAP with our solution – primarily as most production systems have a very small set of ‘robot’ users, and the complexity involved vs using Posix file permissions

We’ve taken this further, and use the virtual hosts of AMQP to provide isolated environments for systems, so configuration managers can partition knowledge of passwords for production and development – and prevent data ‘accidents’.

Management
We provide a rich RESTful API, with language bindings, to give users the management information they need, e.g. queue depth. Of course, how they use it is up to you. One of our customers integrated it with Nagios with a couple of lines of bash. Another uses AMQP’s message timestamps to calculate ‘how stale is my data’ and warn appropriately.

Monitoring
Internally, we monitor our own systems using AMQP. SNMP is too painful (and riddled with security holes), syslog too insecure, and JMX is just… complex, heavy and without a standard transport layer. Our message formats are simple JSON – it’s easy to parse, easy to create and open to every language. One of the beauties of using AMQP for monitoring is adding resource (which happens regularly as a cloud service) requires no changes – there’s just a new publisher.

Summary
So you can see that when we are asked how safe and secure is my data, the answer is very and probably more then many companies internal data centres!

Share

Why is Telemetry a good candidate sector for Message Queuing?

May 25, 2011 – 6:03 am

Telemetry has for a long time been associated with Formal One racing where large banks of software engineers pour over screens as the cars scream round circuits and interpret data as it arrives real time from the cars to decide crucial events such as when to change tyres, when and how much to refill, to advise changes in driving style and the list is endless.

In today’s telemetry markets where anything from water meters to missiles and phones all have capabilities to capture and send data for interpretation to a centralised source the possible long term uses are endless.

Historically however, some limitations have been to do with wireless protocols and most commonly used is SMS. The issues really focus on the inability to ensure that Messages are delivered and not lost which can be critical for some applications. The other issue is also one of pure scalability, SMS is fine for certain applications but when potentially several thousand messages are being sent each second; SMS is unlikely to be able to cope.

Message Queuing as a concept, is ideal for handling large volumes of small amounts of data or messages. It also means that by definition, any data sent by a remote device to a centralised server that may or may not be ready to receive, will not be lost, rather the data will be stored (or queued) on a separate server until the recipient server is ready to receive. In some cases the delay may only be a few seconds or in other situations perhaps a few days. Either way, Message Queuing is ideal to handle such situations.

Moving a step further, historically Message Queuing – while very well proven in concept (having been used by investment banks for the last 15 years or so) has generally been too expensive for all but the largest of companies and applications that warrant such a cost.

However with the open standard of AMQP and StormMQ’s mission to have a Message Queue standard available to anyone as a pure and affordable pay as you go service; means that far more companies who don’t want the hassle of investing in infrastructure and non core skills can simply tap into our Cloud based Message Queue and be part of a hugely scalable and secure system designed not to lose and data at anytime.

This is perfect for many companies now beginning to look at mobile devices as a way of capturing lots of data to be interpreted for competitive advantage. They don’t need to become Message Queue experts or keep within the limitations of SMS solutions. Instead they can sign up with StormMQ, know they won’t be locked in (because it’s an open protocol) and have a totally safe, secure and scalable cloud based service.

The future of messaging is out there – and it’s StormMQ

Share

Announcing libamqp, an open source C client for AMQP 1.0

May 19, 2011 – 11:24 pm

Today, we are very happy to announce libamqp, a C client for the AMQP 1.0 protocol and released under the open source Apache License.

The wide adoption of the AMQP 1.0 specification depends upon fast and reliable open source clients. Eamon Walshe, our CTO started working on such a client a few weeks ago and while his work is still unfinished, today we have released the source code of this initial work.

Eamon has written C Clients in the past for AMQP. In a previous post he wrote about how he developed a client that supports the 0.8 and 0.9.1 versions of the specification for an embedded device. libamqp is a complete departure from this work and only supports the imment 1.0 specification.

By releasing libamqp under the Apache License we are asking the AMQP developer community to join us and contibute to the project. Once this C Client is complete we want to engage developers in the Ruby, perl, PHP, python and other various langage communities to write wrappers too. If you are interested in getting involved please join the libamqp mailing list or contribute at github.

Share

Some recent questions from customers and partners

March 8, 2011 – 11:02 am

We often get asked some of the following questions:

Why did you build StormMQ around AMQP?

We decided to build StormMQ around the emerging AMQP standard because it’s extremely reliable, flexible (in terms of messaging patterns) and it can be used for many messaging scenarios. It scales from simple processes interacting on one machine all the way up to huge, high-bandwidth set ups of different vendors systems talking across the internet.

What is all this about interoperability… are we going to be locked in StormMQ?

We really like AMQP because it has strict rules for the behavior of the messaging provider (StormMQ) and client (your software components). This gives everybody advantages over what’s gone before, the most important of which is that AMQP is completely interoperable… you are not locked into one supplier for the server or client (in just the same way that SMTP, HTTP, FTP, etc. have created interoperable systems).

As a working group, AMQP is completely committed to interoperability…so much so that we have organised 4 connect-a-thons between the various AMQP implementers so that we can ensure that this happens.

So, it’s like JMS?

Certainly not, unlike JMS, which merely defines an API, AMQP is a wire-level protocol. A wire-level protocol is a description of the format of the data that is sent across the network as a stream of octets. Consequently any tool that can create and interpret messages that conform to this data format can interoperate with any other compliant tool irrespective of the implementation language.

What is the difference between XMPP and AMQP?

The differences between XMPP and AMQP are not obvious at first. XMPP is often mistaken as a message queue, especially because of the ability to send “offline chats” and various “targeted message” abilities. Importantly, AMQP supports all of this, and so much more, such as real database-like transactions and deep security, without some of the really annoying stuff such as roster management.

XMPP was designed as a one-on-one communication protocol. You can add extensions to allow things like “broadcasting” but this not supported by default. AMQP supports one-on-one and one-to-many communication through its clever message/exchange system.

I have an existing messaging solution, should I move to StormMQ?

In most cases I would be very reluctant to replace your existing solution if you’re already happy as it’s unlikely to give you a return on your investment. An upgrade would only seem necessary if you have one of the following needs:-

  • You need to interoperate with other systems;
  • You need to talk to systems written in more than one programming language (eg you’re using JMS or MSMQ);
  • The current solution is expensive (in terms of licence fees, consultancy or time investment to maintain);
  • You want to outsource administration, maintenance and support to reduce costs;
  • You can’t scale out easily;
  • The performance isn’t good enough

If the answer to any of these questions is yes then you should investigate the a move to StormMQ.

Share

The road to writing a new protocol is long and lonely

February 24, 2011 – 10:13 am

The AMQP Implements SIG

We have been working on the 1.0 specification for AMQP for around 6 months and for most of that time Raph (our MD) and Eamon (our CTO) have been locked away from civilisation in a darkened room working on code.

Writing a new protocol specification has three main steps, namely the agreement, implementation and interoperability stages. The agreement stage was achieved in May 2010. While this was a very important stage in the protocol development process it is also one of the easiest (in terms of technical challenges). At this stage the protocol is nothing more than a stated objective and because nobody has written a line of code yet it is completely untested.

The implementation stage is a technically complicated stage. StormMQ joined the working group at the start of this stage and we are now working with a team of implementers on code using the AMQP 1.0 spec. This group includes Microsoft, JP Morgan Chase, INETCO, RedHat and RabbitMQ (VMWARE). For the last 6 months this team has been working diligently with only a mailing list to keep the group together. The work has been hard and change to the specification has been frantic. To help keep the process moving we have resorted to organising 4 ‘connect-a-thons’ where the various teams get together for a week to thrash out our issues. The first of these connect-a-thons was in December 2010 in Redmond. RabbitMQ (VMWARE) hosted the second ‘connect-a-thon’  in February 2011 at their London offices. The third event will take place in March 2011 in Boston and the final in May 2011 in Gateshead.

Our goal with this stage of the protocol development is to come up with some real working code examples of how to implement the new AMQP 1.0 specification. Out of this we expect to be well placed to start the interoperability testing…..our goal is to get all this done by the end of July 2011.

In his January 2011 report on the messaging industry, Roy Schulte from Gartner commented that AMQP currently lacks a ‘critical mass’ of vendors. The combined market capitalisation of the companies currently working on AMQP 1.0 code tops US$500b…I consider this a critical mass.

Share