The StormMQ Company Blog

StormMQ and Kaazing bring the MQ to the browser

Today, we are delighted to announce a partnership between StormMQ and Kaazing; the leading innovator in HTML5 WebSocket technology. We are now in a position to provide, as a Managed Service, real time information to the browser thus allowing a one stop shop for end to end Message Queue to browser deployment – without the need for any investment in IT infrastructure support or consultancy.

In the past we were consigned to some horrible HTTP poling hacks…the sort that would keep you awake at night and remind you how poor your developers tool box really is! Gone are the days when a concoction of ajax, xmpp and Comet were stuck together with preverbal sticky-tape to give your web application some resemblance of ‘real time’. With HTML5 web sockets your browser app can interact directly in full duplex mode with your applicaiton or message queue.

Why is this so important? Well for certain applications, being able to have the “richness” of experience as if your PC or mobile device was directly attached to the server can be vital. For example; in trading environments where instant changes need to be shown immediately so that trades can be made based on the absolute latest real time information is crucial.

Without WebSocket technology, organizations requiring instant real time information had to use a hotchpotch of technologies such as ajax, comet, xmpp and TCP IP to deliver their service. This cocktail of plugins reduce flexibility of browser deployment and limits competitive edge when it comes to designing new applications.

The StormMQ cloud based managed messaging solution, integrated with the Kaazing WebSocket Gateway, is ideal for businesses operating in industries with very large customer bases that require real-time information exchange to remain competitive, including financial services, telecommunications and online gaming.

Adherence to AMQP is a given
StormMQ in addition to Kaazing, provided as a subscription-based message queuing platform organizations a low-cost cloud-based alternative to expensive, proprietary messaging middleware.  Combined with Kaazing’s WebSocket Gateway technology, a HTML5-based platform that enables full-duplex Web communication, StormMQ can now extend its service offering over the Web, securely reaching customers wherever they are in the world via a device of their choice such as desktops, Web browsers, smart phones and tablet PCs.

Finally you can sleep without fear of the monsters in your application.

Posted in Announcements | 1 Comment »

Download our webinar

We ran a webinar today using the Webex system which acted as a low level introduction to message queue technology and why you might use an AMQP based message queue. In this webinar we covered some background information, architecture considerations and some use-cases for MQ’s in your organisation.

If you would like to watch this webinar you can download the file here:
http://blog2.stormmq.com/files/2011/02/an_introduction_to_AMQP.arf

To watch this webinar you will need to install the Webex player. If you have not got this installed you will need to download this here:
http://www.webex.com/downloadplayer.html

If you would like to be notified of future webinars please enter your details here:

Posted in Events & Presentations | No Comments »

Apples and Oranges: Security, Authenticity and Queue Permissions

This is part 6 of a 6 part blog series where our MD Raph Cohn attempts to answer our most often asked question…Is StormMQ the same as Amazon SQS? In this part Raph deals with perhaps the most important aspect of any enterprise architecture…security.

Security, Authenticity and Queue Permissions
Every messaging systems offers a different approach and trade offs for security, authentication and permissions. Ultimately, the choice of what’s needed dictates what’s right. Security starts with the transport layer.

Transport Security
Amazon SQS and StormMQ both use TLS to securely authenticate connections and transport message data encrypted to and fro. Importantly, StormMQ is secure by default; you can not disable TLS, whereas, with some versions of Amazon SQS, you can use HTTP for a faster response. With StormMQ, that’s not needed – we can deliver more than 10,000 messages a second, with TLS. However, the initial stages of TLS are plain, and reveal information about the client that might be best kept secret. For the very security conscious, we offer IPSec VPNs to those clients that request them – or we can even run our endpoints inside your data centre (a hybrid cloud model). However, to maintain the privacy of your messages, we go much further

Server Security
At StormMQ, we take the view that the less trust we have to place in things, the better. Consequently, we run IPSec VPNs between all our internal servers, so that no ‘traffic knowledge’ leaks out. Aggressive firewall policies (during testing we even once locked ourselves out of our own servers) prevent non-VPN traffic entering a box. And any messages needing to be temporarily stored on disk are encrypted – with AES 256 bit keys and appropriate cipher block modes. We’ve internally banned use of MD5 and SHA1 – hash algorithms still used in Amazon SQS for successful message enqueues.

Permissions
All these security arrangements at Amazon SQS and StormMQ are not worth a fig leaf, though, without proper user permissions. Like Amazon, we strongly authenticate API calls and server connections, and allow sharing of queues. However, we go much further; we allow different users fine grained read, write and create permissions, with a flexible means of defining which queues (and other important things) they get them on. With Amazon, to set up a secure solutions with different senders and receivers requires separate user accounts and the complexities of more billing and account management. With StormMQ, one configuration manager can define who gets what. Additionally, our command line tools let server or desktop administrators prevent user access to passwords and credentials. With StormMQ’s concept of environments, developers can not accidentally use production or vice versa – thereby saving your bacon on a SAS 70 Type II audit.

Through this Apples and Oranges series I have tried to outlined a few significant differences between StormMQ and Amazon SQS. I have not tried to compare these services on price, only the features and benefits. If you have any questions I would be happy to answer these by email, phone or openlyon this blog.

This blog post is the sixth of six articles written by Raph Cohn. You can see the full Apples and Oranges series here.

Posted in Apples and Oranges | 1 Comment »

Apples and Oranges: Message Properties

Posted in Apples and Oranges | No Comments »

Apples and Oranges: Messaging Idioms

This is part 4 of a 6 part blog series where our MD Raph Cohn attempts to answer our most often asked question…Is StormMQ the same as Amazon SQS? In this part Raph deals with some simple enterprise message queue idioms and which ones are supported by StormMQ and which ones are not supported by Amazon SQS.

Messaging Idioms

A messaging idiom is a typical way of sending data using queues. There are several common idioms, including:-

• Point-to-Point
• Store-and-Forward
• Fire-and-Forget
• Fanout
• Publish-Subscribe (“Pub-Sub”, sometimes known in JMS as Topics)
• Priority Queues

Amazon SQS and StormMQ both support basic ‘point-to-point‘ messaging with ‘Store-and-Forward‘ and ‘Fire-and-Forget‘. This is where a message is created and posted to a queue by a producer, and a consumer then reads it off latter on. Messages are read in the order they’re received. For many applications, such as one system talking to another (eg transferring an end-of-day balance sheet) this is ideal. However, the disadvantage is that the producer has to know something about the consumer – namely, where he looks for his messages.

StormMQ supports more complex idioms using routing. Sometimes, a producer wants all potential consumers to get his message. He doesn’t know who they are. Indeed, the list might change. An example might be broadcasting an alert, or sending heartbeats to servers. Fanout messaging is used for this. One message is sent to everyone interested, on multiple queues, in one go. This can be done with Amazon SQS, but would additional knowledge in the client (eg a database). The whole point of fanout is the client is dumb, and doesn’t need to know – it makes maintenance much easier, as producer and consumer are separated.

A more involved idiom still is publish-subscribe, often just called “pub-sub“. For this, a producer is sending messages, but not every consumer wants every messages. It’s like fanout but with picky users. A classic example might be a system sending out change requests or currency prices. For example, a manager might want to see all change requests, but a development team just gets those relevant to their project. Or some financial traders are betting on currency rates. Some want to know about dollar-sterling, others euro-sterling and others both. And there interests change over time. Doing this with Amazon SQS is fiendishly hard, and rather error prone. It’d need database lookups, application logic and a complex queue setup – especially if different queue permissions were needed. On the other hand, it’s pretty simple with StormMQ.

Last, but no means least, is priority queuing. A priority queue is one which sorts messages by their urgency. When a message is requested by a consumer, the most urgent is given – this might not be the one that is oldest. These sorts of queues are used for call dispatch systems or ambulance HQs – typically, ‘end-user’ systems. StormMQ has simple priority queues. However, systems using them often have more complex logic for deciding priority (eg urgents first, but occasionally, an important if it’s over 10 mins) and so care needs to be taken in deciding the right queuing approach and solution. With such systems, uptime, reliability and data protection often matter more – something StormMQ is ideal for.

In terms of enterprise functionality, Amazon SQS does not compare to StormMQ.

This blog post is the fourth of six articles written by Raph Cohn. You can see the full Apples and Oranges series here.

Posted in Apples and Oranges | No Comments »

Apples and Oranges: Acknowledgements and Transactions in Amazon SQS and StormMQ

This is part 3 of a 6 part blog series where our MD Raph Cohn attempts to answer our most often asked question…Is StormMQ the same as Amazon SQS? In this part Raph deals with Acknowledgments and Transactions….these are the features that seperate enterprise messaging systems and Amazon SQS.

Acknowledgements and Transactions in Amazon SQS and StormMQ

Amazon SQS is a superb product which implements a simple messaging API. One of its advantages is a straightforward approach to consuming messages: Consume a message, and within a time frame, acknowledge you’ve got it. If not acknowledged (‘acked’), put it back on the queue. StormMQ, using AMQP, provides a very similar model. If you’re just occasionally pulling a message off a queue (ie, polling for new messages) then there’s little more to say: they’re both simple and unfussy. However, there are common situations when simple acknowledgements won’t do.

Selective Acknowledgements
Let’s take an example of an application processing a stream of data, such as a series of position changes for the Mars Rover, or a feed of corporate bond data on a Stock Exchange (eg price, sales volume, yield to maturity, Z score and PV01, say). A common way of sending such data is a ‘key frame’ or ‘last value’ of all the values followed be a series of messages just detailing changes – it keeps data volume low (the same technique is used for encoded video). Ideally, you’d want to be pushed new data, rather than go and get it every time – it’s more efficient. Both StormMQ and JMS do this, but Amazon SQS doesn’t. Of course, if the volume or throughput of the stream is quite high, you might not be able to, or might not need to, acknowledge every message. But if your application fails, you would like what hadn’t been finished with to be pushed back on the queue.

That’s something that JMS can only do with heavyweight transactions, but AMQP can do with lightweight selective acknowledgments. Simply tell the server (‘ack’) the unique id of the latest message you handled – and it’ll acknowledge as received and used all previous messages sent to you.

Managing Quality of Service
The messaging experts amongst you might be getting very worried by now. How long should StormMQ’s server wait (either in time or number of messages) before assuming something bad happened? What if we don’t ack for 1000 messages and the client gets swamped? That doesn’t sound good! That’s where AMQP’s Quality of Service can be used – simply tell the server how many messages it can send before waiting for an ack. Used judiciously, this can become a very simple and effective way of having a client cache incoming messages, effectively smoothing out the profile of that stream of data, so the valuable application logic you’d write on top is fully utilised. Trying to do the same with JMS transactions will consume resources like mad and slow down the whole set up for everybody.

Turning it all Off
Of course, sometimes acknowledgements just plain get in the way. Imagine you’re writing an application to display the latest status of something – perhaps a monitoring app displaying log messages in near realtime, a call centre ‘calls handled’ display, or a small ‘latest stock price’ app. It doesn’t really matter if you don’t process every message – another one will be along in a second or two. Perhaps they come almost too fast – StormMQ can transmit a message from creator to consumer in under 14ms. In that cases you can just turn them off with StormMQ – something you just can’t do with Amazon SQS.

Making sure Messages Go!
It’s often the case in messaging that ensuring a message gets onto a queue matters. A sort of reverse of acknowledgments. The simple way we do this in StormMQ is with transactions. Transactions in AMQP are like those more commonly found in databases, but without a lot of the complexity. They’re designed to make a message (or messages) definitively got on a queue, and that an ack really happened. They exist in JMS, but are absent in Amazon SQS.

The simplest need is to make a message got on a queue. In normal AMQP, a transaction isn’t needed – as long as the connection is still open after posting a message, and no error was reported, it got there – AMQP only reports failures. However, if you’re using a mobile device, that might not be enough. A recent client we worked with was constrained to using a GRPS modem. The connection to StormMQ, over TCP/IP, was initiated by the modem, but happened at the mobile companies data centre. As is the way with mobiles, the GRPS modem would often lose the signal and drop the connection – but only to the data centre. It had no way of knowing when the mobile company’s servers would kill its TCP connection to us – was it before or after the signal was restored? If signal was lost just after sending a message… well, let’s just say with mobiles, if it can happen, it will (and to some, TCP and GRPS doesn’t mix – something us developers have no control over). By using a transaction, though, they could determine delivery.

A more complex example might be a server that publishes several related messages – perhaps the breakdown of a bill, or instructions to a group of systems (perhaps an address change from the CRM to accounts, stock control and procurement). They either all go on the queue or nothing goes on the queue. StormMQ supports that too.

Through this series of blog posts I am trying to show that StormMQ and Amazon SQS are so vastly different they cant even be compared. Amazon SQS is an excellent product, but if you need any level of transaction then it will not work and you need an enterprise message queue…like StormMQ.

This blog post is the third of six articles written by Raph Cohn. You can see the full Apples and Orangesseries here.

Posted in Apples and Oranges | No Comments »

Apples and Oranges: Legal

This is part 2 of a 6 part blog series where our MD Raph Cohn attempts to answer our most often asked question…Is StormMQ the same as Amazon SQS? In this part Raph deals with the legal issues and tries to answer the biggest question in cloud computing…. Dude, Where’s My Data?

Legal Compliance

The words “IT Audit” strike fear in every developer’s heart. If you have ever had an audit you’ll know that they can cause a world of pain. An IT auditor is somebody to be feared. Some auditors are straightforward, and you can ‘get away with a few things’. But many are very smart, very knowledge and do an excellent job. They will let you away with nothing. While being painful, the process is very important for IT departments. It forces them to design better systems which benefit the organisation and the customers.

If you use Amazon SQS, you get a great service. It’s brilliantly engineered, with a simple interface to a complex architecture. But it has one weakness – it is a legal minefield. In the EU, every country has a data protection commissioner. In the UK, it’s the ICO, for example. Each country has national rules, but they all derive from the same EU directives. The essence of this legislation is it makes you responsible for looking after customer and other personally identifiable data even if you use a third party provider or contractor. To pass any IT Audit you need to prove that not only you know the locality of your data, but:-

• that it’s secured in transit;
• encrypted when stored;
• can not be subverted, altered or changed without knowledge, and,
• is only revealed to those that need to know

That’s hard to do. In addition to this there are four key things to consider:

1. Location. Where is the data stored? If you use any cloud based provider you will need to consider this. You need to be VERY careful if you move personally identifiable data from one jurisdiction to another. You might think that choosing the Amazon SQS service in Dublin will side step this issue..and in many cases it might…but remember that they are a US company that is still covered under the US Patriot Act. Ross Cooney, our Commercial Director has a rant about this on his blog.
2. Consistency. There’s no point having data if it’s not consistent. Well, if you want to use Amazon SQS you might want to read clause 11.5 of the Amazon Web Services Customer Agreement, which states:
   WE AND OUR LICENSORS DO NOT WARRANT THAT THE SERVICE OFFERINGS WILL FUNCTION AS DESCRIBED, WILL BE UNINTERRUPTED OR ERROR FREE, OR FREE OF HARMFUL COMPONENTS…..
   You will not pass an IT audit with this sort of a statement in the customer agreement.
3. Security. The clause listed above goes on to say:
   …OR THAT THE DATA YOU STORE WITHIN THE SERVICE OFFERINGS WILL BE SECURE OR NOT OTHERWISE LOST OR DAMAGED.
   So, if Amazon is hacked then not only will you have to apologise to your customers you will also have to explain to the Data Commissioner why you used a service that was insecure. The Data Commissioner can hit you with a £500,000 fine for this sort of thing. Are you scared yet?
4. Persistence. How long is your data on their system after you delete it? Are all the copies destroyed? Is it still there? Dude, where’s my data?

At StormMQ, we care deeply about data security and we can help you demonstrate audit compliance. We use 2048-bit TLS and IPSec VPNs to secure access – even on our own network. We use encrypted disk drives and UK based ISO accredited data centres with 24 hour manned security. StormMQ was built from the ground up as a service that compiles with as many regulatory requirements as possible.

Locating and securing your data matters to us. Our optional Locate-It^tm service will sign-over ownership of the encrypted hard disks used to hold your data. We will then provide you with a Certificate of Locality and Ownership of Data. Using these certificates you can have your auditor or IT director check our facilities and audit the actual hardware that is used by your message queue. Ownership of the hard disks also gives you the added security of being able to take the disks at the end of your subscription for secure destruction.

Our service does not compare our service to Amazon SQS when you consider the legal implications. But, dont take our word for it, we strongly recommend that you consult your lawyers.

This blog post is the second of six articles written by Raph Cohn. You can see the full Apples and Oranges series here.

Posted in Apples and Oranges | 2 Comments »

Apples and Oranges: Performance

We are often asked to explain the difference between our StormMQ service and the Amazon Web Services SQS service. The reality is that we are significantly different, it’s like comparing apples and oranges.

After a recent team meeting our MD Raph Cohn sat down and wrote six articles that describe these differences. Over the next few days I will post these articles on this blog for you to read. They cover the following topics:

• Performance
• Legal
• Acknowledgments and Transactions
• Messaging Idioms
• Message Properties
• Security, Authenticity and Queue Permissions

The first article is here:

Performance
Performance. One of those words that comes up time and again in IT, is misused as much as understood and seems to drive decisions a bit too much. In many ways, performance doesn’t matter – at least not to start with. Getting it right matters far more. In that respect, it doesn’t matter if you choose Amazon SQS over StormMQ; getting your application out does. However, an understanding of what you’re trying to does.

Amazon SQS is ideal if you:

• need to send and process a few small messages a day.
• send data that isn’t covered by legal compliance issues
• send data that can be occasionally lost
• send data where retrieval and processing time isn’t critical
• want to use a HTTP based API, is for use from inside a web page or java script client.

If your application can work under these requirements then StormMQ would be a poor choice…basically because Amazon SQS will cost you only a few dollars a month.

However, when transit time is important (‘message latency’), then Amazon SQS starts to look a little different. Using Amazon SQS you can send a message in under a second if you are lucky but it often takes a few seconds. Then when you go to retrieve the message you need to use an incredibly inefficient HTTP poll that wastes bandwidth and CPU resource.

With StormMQ we can turn a message around in under 14ms. However, it can seem faster, as we don’t use an HTTP connection for a new message and another one for an ack. Indeed, using a consumer, messages just arrive in the background, for processing. So if you’ve got a long lived app, and need to take action as soon as something arrives, we’re ideal. If you’ve got multiple consumers, we get to become far more efficient, automatically pushing messages where they’re needed.

In terms of performance, StormMQ and Amazon SQS dont compare.

This blog post is the first of six articles written by Raph Cohn. You can see the full Apples and Oranges series here.

Posted in Apples and Oranges | No Comments »

StormMQ join the AMQP Working Group

StormMQ is built around the AMQP protocol, a 4 or 5 year old protocol that was designed when a group of vendors got together with a few users with a vision to build a new, high speed, reliable and open protocol for the MQ market. The goal was to produce the next generation MQ protocol to replace the incumbent proprietary standards. Over the past few years this protocol has passed through a few iterations but the current agreed standard is 0-9-1 and it is this standard that StormMQ currently adheres to.

For the last few months the AMQP Working Group have been working diligently on the AMQP 1-0 specification. We decided at StormMQ that we wanted to contribute to this in an active role in the Working Group and made an application last month. After several interviews and plenty of due-diligence we were formally accepted into the Working Group today (5th January 2011).

We look forward to working closely with the other Working Group members to get AMQP 1-0 agreed and released. Soon after this specification has been agreed we plan to release a new version of StormMQ that supports this new specification.

Other members of the Working Group are:

• Bank of America, N.A.
• Barclays Bank PLC
• Cisco Systems, Inc.
• Credit Suisse
• Deutsche Börse Systems
• Goldman Sachs
• INETCO Systems Limited
• Informatica Corporation
• JPMorgan Chase Bank & Co.
• Microsoft Corporation
• Novell
• Progress Software
• Rabbit Technologies (part of VMWare)
• Red Hat, Inc.
• Software AG
• Solace Systems, Inc.
• Tervela, Inc.
• TWIST Process Innovations
• VMware, Inc.
• WSO2, Inc.
• 29West Inc. (part of Informatica)

Tags: AMQP, amqp working group, Stormmq

Posted in Uncategorized | 2 Comments »

AMQP 1-0 and the AMQP Connect-a-thon

Last month our development team at StormMQ started implementing AMQP 1-0 support in our service. The AMQP 1-0 spec is a big departure from the existing 0-9-1 spec and we expect to have a market ready version in early Q2 2011….but this all depends on the full release of the AMQP 1-0 specification from the AMQP Working Group.

In an effort to expedite the release of the AMQP 1-0 specification th AMQP Working Group organised a tech meetup called a ‘connect-a-thon’. The AMQP spec is designed to be interoperable, so in an effort to encourage this the Working Group asked anybody who is developing an application using the spec to meet up and start interoperability testing. Using this experience we will be able to help the Working Group iron out any remaining issues with the specification…this being a crucial step on the way to the final version of AMQP 1-0.

Last week I flew to Redmond, Washington in the USA to take part in the connect-a-thon. The event was hosted by David Ingham from Microsoft and was attended by Redhat, JPMC, INECO, Vmware (RabbitMQ) and Microsoft.

At the event I revealed our first AMQP 1-0 modules and a ‘codec’ that allows you to translate from a text form to a binary form and back again using the AMQP 1-0 spec. I found it very useful to test this codec against all sorts of test data and then compare the results against other vendors.

Once we get a public BETA of our AMQP 1-0 implementation I will announce it here.

Tags: AMQP, AMQP 1-0, connect-a-thon

Posted in Events & Presentations | 1 Comment »