Introduction

Today’s mobile devices represent exceptional foundations for wide-area Internet of Things (IoT) applications. However, the vast heterogeneity of real-world environments makes it challenging for applications to sense the user’s evolving context and intelligently interact with smart-objects. To address these challenges, we’re developing Ambient Dynamix (Dynamix): an open plug-and-play context framework for Android. Dynamix is a community-based approach for context-aware computing, where advanced context sensing and acting capabilities are deployed on-demand to mobile devices as plug-ins, and are made available to applications through only a few lines of code. Dynamix runs as lightweight background service on the user’s mobile device, leveraging the device itself as a sensing, processing and communications platform. Both native and web-based mobile applications request context support from Dynamix using simple application programming interfaces (APIs). Dynamix automatically discovers, downloads and installs the plug-ins needed for a given context interaction task. When the user changes environments, new or updated plug-ins can be deployed to the device at runtime, without the need to restart the application or framework. Dynamix comes with a growing collection of ready-made plug-ins, and provides open software developments kits (SDKs) and a scalable repository architecture, which enable 3rd party developers to quickly create and share new plug-ins types with the community.

Since this video was made, Dynamix has been updated to support actuation with local and remote computing resources in addition to context sensing. Dynamix also now supports Web applications in addition to native Android apps. Please see the documentation below for details.

Architecture

The Android platform was chosen as the initial deployment target for the Dynamix Framework, since it provides rich capabilities such as background processing, dynamic code linking, and a comprehensive inter-process communication (IPC) model. Dynamix is designed as an Android Service that runs in the background providing context-awareness services to (potentially many) Android applications. The Dynamix Framework operates according to the Android Service Lifecycle and is kept alive as long as there are sufficient device resources (and restarted when possible). Although Dynamix generally serves applications invisibly in the background, users can configure its settings using an inbuilt graphical user interface, which provides a means of controlling the context firewall, managing permissions, performing updates, adjusting plug-in settings, etc. Dynamix also alerts users of important status changes using Android’s notification system.

As shown in Figure 1, a Dynamix Service is situated between a device’s local hardware and (potentially many) Dynamix apps, which execute in their own runtime processes. Dynamix supports two principal app types: Native apps and Web apps. Native apps are defined as standard Android applications that communicate with a local Dynamix service using its AIDL-based application programming interfaces (APIs), which are included in the App SDK. These interfaces include an AIDL Facade API, which enables apps to request and control context support; and an AIDL Event API, which enables apps to receive framework notifications and context events. Web apps are hosted by native Web browsers, such a Google Chrome and Firefox. To support communication with browser-based Web apps, Dynamix exposes two local REST-based APIs using a customized Web server that is embedded within the Dynamix Service. Web apps communicate with Dynamix via Ajax, using two provided JavaScript support files that simplify service binding, API interactions, data serialization/deserialization, error handling and event processing. Web app support is discussed in detail below.

DynamixFrameworkArchitecure5

Figure 1: Overview of the Dynamix Framework

Context interactions (e.g., sensing or actuation) are performed by the Dynamix Service on behalf of apps using a tailored set of Context Plug-ins are created as standard OSGi bundles, which are packaged and deployed as Java Archive (JAR) files with additional OSGi metadata. Context Plug-ins can be discovered, downloaded and installed into Dynamix’s embedded OSGi container at runtime from a configurable set of public or private repositories (network and filesystem). Plug-ins expose simple, high-level APIs to applications, which insulates app developers from the domain complexities common to context-aware computing, such as machine learning; sensor fusion; time-of-flight calculations; image processing; etc. To manage Context Plug-ins internally, the Dynamix Framework utilizes an industry standard OSGi container, which is used to dynamically integrate mobile code, provide secure access to local resources, impose a comprehensive plug-in life-cycle model (including thread priority), handle dependency management and enforce plug-in versioning. Using its embedded OSGi container, Dynamix is capable of downloading, installing (or updating) and integrating Context Plug-in Bundles during runtime without restarting the framework. Context Plug-ins are loaded into the Dynamix Framework’s runtime process (with additional security constraints) and operate in accordance with a tightly controlled lifecycle. Private repositories can be configured by users, and the Dynamix project hosts a public plug-in repository using a load-balanced, high-availability server architecture. To reduce server-side complexity, client-side logic built into the Dynamix Service provides plug-in discovery services, including local capability analysis, plug-in filtering (based on local capabilities and plug-in requirements) and plug-in installation.

The Dynamix Service automatically manages the power state of all context plug-ins by notifying them about the user’s power preferences during initialization, runtime (e.g., the user selects the power saver profile), and during relevant Android system events (e.g., screen on/off). Additionally, context events are (optionally) cached by the framework’s Context Cache, which helps applications discover past events and lowers the resource burden on the device by intelligently resending cached events rather than forcing Dynamix to continually re-request (often computationally expensive) context scans.

Context Events

As plug-ins interact with the physical world, they may send context information to apps as context events. Context events are securely provisioned to apps based on the security policies defined by the user in the Dynamix Context Firewall (discussed shortly). Plug-ins encode context events using Plain Old Java Objects (POJOs), which enable apps to work with objects-based context representations rather than potentially cumbersome strings. Plug-ins can be manually installed into a Dynamix Service or deployed automatically in the background in response to app requests. Figure 2 shows the Dynamix Service’s Home tab (left) and the Plug-ins tab (right) during a plug-in installation process.

plugin-install

Figure 2: The Dynamix Service’s Home and Plug-ins Tabs

Context Firewall

As context events may contain sensitive information, Dynamix provides an integrated Context Firewall that securely provisions events to requesting apps based on the security policies defined by the user. The Context Firewall enables end-users to precisely manage the context information available to each application. Unlike traditional Android app security permissions, which must be accepted at install time and cannot be changed afterwards, the Context Firewall can be adjusted at any time. This enables users to dynamically update an app’s privacy settings, block specific apps, or disable context support entirely. Context Firewall settings are defined per application using preconfigured Privacy Policies and/or Custom Privacy Settings. Apps are blocked from accessing plug-ins or receiving any context information from Dynamix by default. During the first interaction with an app or website, Dynamix informs the user through Android notifications (optionally using audible sounds and vibration) who can then specify privacy settings for the app. To define a Context Firewall policy, users click on the Android notification icon or open the Dynamix Service’s control interface. Figure 3 shows the definition of a Context Firewall policy for the sample Dynamix Logger application.

context-firewall-1

Figure 3: Defining a Context Firewall Policy

A Privacy Policy assigns an overall trust level to an application, ranging from blocked to highest. To allow an app to receive context events at a particular privacy risk level, users must specifically grant an app an appropriate trust level (using a Privacy Policy) or apply Custom Privacy Settings. A Privacy Policy automatically assigns an associated trust level to each installed context plug-in, which determines the privacy risk level of the context information that is allowed to flow to the app (e.g., the “Low Trust” policy allows only low risk context information to pass from the plug-in to the application). For more fine-grained control, Custom Privacy Settings can be assigned to a particular plug-in, which override the privacy risk level assigned by the Privacy Policy. For example, an app may be assigned a Medium Trust policy, but be blocked from receiving positioning information from a particular plug-in. To assist the user in selecting proper privacy settings, each context plug-in developer provides a detailed description of the privacy risk levels supported by a given plug-in, as shown in Figure 4.

context-firewall-2

Figure 4: Tapping a Plug-in Reveals its Privacy Risk Levels

Context plug-ins interact directly with the device’s underlying capabilities within a Plug-in Security Sandbox, which provides secured access to system resources, user data and Android services. The plug-in sandbox is being developed using a combination of OSGi security, a custom Java security manager, and a secured version of the Android Context (to control access to system resources, such as sensor managers). A complimentary plug-in permission model is also being developed to allow users to control (and modify) a plug-in’s access to system resources. All interactions between the Dynamix Service and plug-ins are multi-threaded (to prevent blocking) and are monitored for timely completion.

Web App Support

In addition to serving native apps, Dynamix can support conventional Web apps within popular mobile Web browsers, such a Google Chrome and Firefox. To support communication with browser-based Web apps, Dynamix exposes two local REST-based APIs using a customized Web server that is embedded within the Dynamix Service. Web apps communicate with Dynamix via Ajax, using two provided JavaScript support files that simplify service binding, API interactions, data serialization/deserialization, error handling and event processing. The embedded Web server listens for requests from Web apps on the localhost and utilizes an associated REST API Manager (REST Manager) to translate requests into standard Dynamix method calls (and vice versa).

AmbientWebArchitecure1

The AmbientWeb Dynamix Extension

As shown above, the Dynamix Framework was extended with a set of REST-based APIs, which are exposed using a customized Web server that is embedded within the Dynamix Service. The embedded Web server listens for requests from Web apps on the localhost and utilizes an associated REST API Manager (REST Manager) to translate requests into standard Dynamix method calls (and vice versa). The embedded Web server and REST Manager are multithreaded in order to support multiple Dynamix listeners and browsers simultaneously. As AmbientWeb is built on top of the Dynamix Framework, Web apps can access Dynamix’s comprehensive set of context features, including dynamic plug-in discovery and installation, adaptive context sensing and acting, event caching, power management, etc.

AmbientWeb apps interact with the local Dynamix Service using two provided JavaScript files: dynamix.js and listener.js. Together, these files simplify AmbientWeb app development by encapsulating Dynamix binding, REST API interactions, state management, data serialization/deserialization, error handling and context event pre-processing. The dynamix.js file provides a complete implementation of the Dynamix Façade API, including methods for adding listeners; setting up context subscriptions; requesting context information (e.g., heart-rate, indoor positioning); and performing ad-hoc interactions with local or remote computing resources (e.g., controlling home automation equipment). The listener.js file provides a complete implementation of the Dynamix Event API, including callback methods for context events, context subscription updates, framework state information, plug-in installation status, etc. To simplify client-side programming, methods in the listener.js file dynamically populate incoming context events with JavaScript properties that enable Web apps to access context data without string-based parsing (string-based encodings are also supported).

To provide a seamless user experience, interactions between AmbientWeb apps and local Dynamix Service are based on a combination of synchronous and asynchronous Ajax. Broadly, Ajax is a group of client-side Web technologies that enable Web apps to request additional data in the background without needing to refresh the current page or interfere with ongoing processing and user interactions. Synchronous Ajax is used to immediately return status messages for Façade API method calls (e.g., success or fail), whereas asynchronous Ajax is used to return data for long-running operations, such as context scans (via the Event API).

AmbientWeb apps are constructed using standard Web technologies (HTTP, CSS, JavaScript) and are deployed using Internet-based Web servers. To create an AmbientWeb app, a Web developer downloads the previously introduced JavaScript files from the Dynamix website and integrates them into a conventional Web app using HTML <script> tags. The developer also browses the available plug-ins listed on the Dynamix Website and notes the identifiers of the plug-ins needed. Next, the developer creates the Web app using conventional Web technologies. When context support is needed by the Web app, functions of the dynamix.js file are used to bind the Web app to the Dynamix Service and establish a Dynamix listener to receive asynchronous callbacks. Next, the developer uses functions of the dynamix.js file to create context subscriptions, control context scanning and/or interact with local and remote devices. Façade API requests are automatically routed from the Web app over the local network to the Dynamix Service, which performs the necessary context interactions on behalf of the Web app. Resulting events and status updates are sent from the Dynamix Service back to callback methods in the Web app’s listener.js file, which can be easily hooked into the Web app’s logic as needed (e.g., adapting behavior or displaying information).

As AmbientWeb apps are loaded into unmodified browsers, they are subject to the same-origin policy (SOP), which is a commonly implemented security restriction that prevents Web apps from accessing the data or methods of scripts loaded from anywhere other than the Web app’s origin (scheme, host, and port). Hence, by default, a Web app is only able to make Ajax requests to the domain it was loaded from, and cannot access the Dynamix Service. To overcome the SOP restriction, Dynamix’s REST Manager is configured to support Cross-Origin Resource Sharing (CORS). CORS is a specification that enables secured cross-site HTTP access between a Web app loaded from one origin (e.g., http://ambientdyanmix.org) and a Web server running in a second origin (e.g., http://localhost). In Dynamix, we configure the embedded Web server with a CORS policy that permits interactions between Web apps and the REST Manager over the localhost.

The REST Manager securely identifies Web apps using the origin and referrer URLs contained within the HTTP headers provided by the browser. To prevent HTTP header spoofing by malicious installable apps posing as Web browsers, Dynamix validates the identity of all apps requesting Dynamix services using AmbientWeb. Identification and verification are accomplished by obtaining the requesting app’s Linux user id (which is specific to applications on Android) by comparing the requesting app’s incoming socket port with the list of open network connections available in the Android Kernel’s /proc/net/tcp and /proc/net/tcp6 virtual file systems. Once a calling app’s user id is identified, the associated app’s X509 certificates are extracted from the Android operating system and compared against a list of approved X509 certificates known by the Dynamix Service. If the calling app is not signed by an approved certificate, Dynamix refuses the request, preventing unauthorized access by unsafe applications. Dynamix includes an updatable set of approved certificates for browsers known to disallow HTTP header spoofing.

When Web apps interact with Dynamix using AmbientWeb, they are subject to the same Context Firewall security restrictions that apply to conventional Dynamix apps (see III). As with conventional Dynamix apps, AmbientWeb apps are blocked from receiving context information from Dynamix by default. During the first interaction with a newly detected AmbientWeb app, Dynamix informs the user through Android notifications (optionally using audible sounds and vibration). The user can then precisely specify which context information should be shared with the AmbientWeb app. Security policies can be defined permanently for a specific Web app or defined only for a single browsing session. A Web app’s security policies may be changed or removed by the user at any time.

Community

We view the Dynamix Framework as the foundation of a broader community initiative that can be used to unlock the wealth of context expertise that is often inaccessible to mobile app developers. With reference to Figure 5, the Dynamix communities are briefly described as follows. First, the infrastructure developer community is responsible for creating the Dynamix framework and the associated plug-in repository architectures. Second, the plug-in developer community wraps domain expertise into reusable context plug-ins and context types, which can be dynamically provisioned to Dynamix-based devices on-demand. Third, the app developer community creates Android-based software based on the Dynamix framework and contributed context plug-ins.

DynamixRoles

Figure 5: Dynamix Developer Communities

To support the application developer community, the Dynamix project provides an open Plug-in SDK, which provides the foundational classes, documentation and samples necessary for 3rd party domain experts to create a wide variety of plug-ins for the Dynamix Framework. As context information is often specific to a particular context domain, the Plug-in SDK also provides the IContextInfo interface, which enables developers to create arbitrarily complex context representations using Plain Old Java Objects (POJOs). IContextInfo implementations are self describing, providing the context type, supported context information encodings and the fully qualified class-name of the implementation. Plug-in developers can create their own implementations of the IContextInfo interface or implement an existing IContextInfo type provided by the Dynamix community.

To support the plug-in developer community, the Dynamix project provides an open App SDK, which provides the foundational classes, documentation and samples necessary for mobile developers to create IoT apps using the Dynamix Framework. When creating a Dynamix app, a developer can select from the growing list of available plug-ins on the Dynamix Website, which describes each plug-in’s identifier, functionality, operational semantics, supported context types, required hardware, etc. After integrating the App SDK’s JAR file into an app, developers use the Dynamix Service’s Facade API to request context support from the local Dynamix Service. The Facade API can be used to access framework state information, install and remove listeners, query available plug-ins, request cached context information, and create context subscriptions. A context subscription is a registration made by a Dynamix app indicating that it wishes to receive context events of a particular type or influence environmental context. During context subscription registration, the requested context type is checked against the plug-ins installed in the local Dynamix Service. If the Dynamix Service is able to support the context subscription type (i.e., it has a compatible context plug-in installed), it sets up the subscription, initiates context sensing and/or interaction, and informs the app using the Event API. In some scenarios, a Dynamix Service may not initially have support for a requested context subscription type. In this case, the Dynamix Service may attempt to discover and install compatible context plug-ins using its configured set of plug-in repositories. If a suitable context plug-in can be found, it is installed during runtime and the context subscription is established automatically. Once context information has been received by the app, it is then free to react as needed, according to its internal application logic.