Architect Quality, Paysafe - Sofia, Bulgaria
Denitsa has more than 15 years of experience as front end, backend and tools developer, trainer and consultant as well as integration engineer and development support. She believes in “quality on each step” of the processes and delivery cycle, from design to support. As Architect Quality at Paysafe she is leading the process of defining the testing strategies, processes and conventions, coordinating efforts across all campuses over the globe.
About her talk
Microservices and BigData Clash: How to Make a Synergy of Opposites
How do we combine two contradictory at first glance approaches: Microservices and Big Data? Are they really mutually exclusive?
Let us look at the key principles of Microservices:
- Single purpose: designed to do only one thing.
- Self-contained: must be able to work as separate program.
- Private database: no shared data schemes and tables. No touching another service data.
- Testing: tested in isolation
- Contract driven: define a contract and stick to it.
And from the other side we have the Big Data solutions: Hadoop, MapR and Google Analytics, which offer huge cluster able to load, contain and share a huge amount of data between multiple producers and consumers. Key principles:
- Shared storage: sharing the same data and information.
- Fast processing:
- High volume: huge files and or multiple data/files should be processed
- High velocity: the data is produced very fast
- High variety: the data is in multiple formats
From one side, we have the Microservice which require a private database and from the other side we have the Big Data solutions that are designed with the purpose of sharing the same data between multiple programs. Making synergy of those two require an answer of the questions: Can we have a microservice working with Big Data solution as a storage? Can we do the testing in isolation? What amount of data is considered Big Data? Does it make sense to test Big Data with small data sets?
You will see our answers to those questions and our implementation of MICROservices and BIGdata. Going through the journey we took in making such system ticking like a Swiss watch and proving its quality as part of continuous delivery on a large scale. You will see where the pit falls and the challenges are and how to avoid and overcome them.