|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Efficient Pub/Sub Storage Mangaement for Big Data
Publish/Subscribe systems allow subscribers to monitor for events of interest generated by publishers. Current
publish/subscribe query systems are efficient when the subscriptions (queries) are relatively static – for instance, the set of followers in
Twitter – or can fit in memory. However, an increasing number of applications in this era of Big Data and Internet of Things (IoT) are
based on a highly dynamic query paradigm, where continuous queries are in the millions and are created and expire in a rate
comparable, or even higher, to that of the data (event) entries. For instance moving objects like airplanes, cars or sensors may
continuously generate measurement data like air pressure or traffic, which are consumed by other moving objects.
In this Project, we propose and compare several publish/subscribe storage architectures, based on the popular NoSQL Log-Structured
Merge Tree (LSM) storage paradigm, to support high-throughput and dynamic publish/subscribe systems. Our methods naturally
support queries on both past and future data, and generate instant notifications, which are desirable properties missing from many
previous systems. Further, we show how hierarchical attributes, such as concept ontologies, can be efficiently supported; for example,
a publication’s topic is “politics” whereas a subscription’s topic is “US politics.” We implemented and experimentally evaluated our
methods on the popular LSM-based LevelDB system, using real datasets. Our results show that we can achieve significantly higher
throughput compared to state-of-the-art baselines.
Acknowledgements
This project is partially supported by NSF grants IIS-1447826 and IIS-1619463.
People
Download
- Pub/Sub Storage Implementations over LevelDB
Publications
Conferences/Workshops/Journals
|
|
|