Dev.Wednesday – Big Data

Four months ago, we started the event series Dev.Wednesday in Bern. Since then, the community has been growing and the event got quite well known. See also our recently published movie about Software Development @Swisscom which was produced at Dev.Wednesday in December. We are happy to announce that the last two events were fully booked in a very short period time. So be fast for next event on April 20, 2016 in Bern 🙂 We will have another hot topic – more about it at the end of this blog post.

One week ago we had the pleasure of having Iulian Dragos from Typesafe as a great and experienced speaker at our fourth Dev.Wednesday. He was giving us an introduction about how you can leverage mesos for running Spark Streaming jobs in production.

Iulian Dragos on stage

Iulian Dragos on stage

What’s mesos and what does it?

Mesos is a general purpose cluster manager that can scale to tens of thousands of nodes and that can handle mixed data loads and general applications. Mesos is being used in large deployments such as Twitter and AirBnB. Its versatility makes it particularly appealing to organizations that have a mixed workload and want to maximize their cluster utilization. Iulian showed us how exactly this manager works when the workload is a long-running Spark Streaming job. Particularly, how does one deal with failures that are bound to happen at this scale, without data loss and service disruptions.


In that talk we:

  • discussed how Spark integrates with Mesos
  • talked about the differences between client and cluster deployments
  • compared and contrasted Mesos with Yarn and standalone mode
  • were looking at deploying a Spark Streaming application that should run 24/7
  • and finally Iulian showed how to deploy, configure and tune a Mesos cluster such that the application runs efficiently and uses only the resources it needs, as well as if any of the nodes fail (including the driver), the application recovers without data loss.


If you are interested in Iulian’s presentation, you can find his slides here.

As you already might know, we always finalize Dev.Wednesday with Beer o’clock because it is a great opportunity to continue discussions with colleagues, new acquaintances and the speaker(s). At this point, thanks again Iulian for coming to our event!

Beer o'clock at Dev.Wednesday

Beer o’clock at Dev.Wednesday

As I said before, our next Dev.Wednesday will take place on April 20, 2016. Our guest speaker is going to be David Watrin, Head of Security and Intelligence at Swisscom and he will talk about the hot topic Blockchain. That means we are going to explore the capabilities and the potential of the Blockchain technologies and what Swisscom’s stakes could be. Seats for the next event are very limited again. So please register at Meetup or attending only if you really want to attend the event. Devs on the waiting list will be thankful 🙂

Selfie of hard core

Obligatory last pic of the “hard core” 🙂