This is a catalog of FIDL examples intended to demonstrate FIDL concepts through simplified implementations of real software workflows.
Example index
The following examples sequentially demonstrate useful FIDL concepts.
Calculator
The calculator example shows fundamental building blocks for creating your first FIDL protocol.
Key-value store
The key-value store example demonstrates how to build a simple key-value store using FIDL in order to learn about the various data types available in the language.
Canvas
The canvas example demonstrates how to build a simple 2D line-rendering canvas using FIDL in order to learn about commonly used data flow patterns.
Concept index
Each "concept" in the FIDL language is exemplified in at least one of the examples listed in the preceding section. A quick reference of each such concept, as well as its example implementations, is listed in the following section.
Acknowledgement pattern
FIDL recipe: Acknowledgement pattern
The acknowledgement pattern is a simple method of flow-control for methods that would otherwise be one way calls. Rather than leaving the method as a one way call, it is instead turned into the a two way call with an absent response, colloquially known as an ack. The ack's only reason for existence is to inform the sended that the message has been received, which the sender can use to make decisions about how to proceed.
The cost of this acknowledgement is added chatter over the channel. This pattern can also result in degraded performance if the client waits for the acknowledgement before proceeding with the next call.
Sending unmetered one way calls back and forth produces a simple design, but there are potential pitfalls: what if the server is much slower at processing updates than the client sends them? For example, the client may load a drawing consisting of many thousands of lines from some text file, and try to send them all sequentially. How can we apply back pressure to the client to prevent the server from being overwhelmed by this wave of updates?
By using the acknowledgement pattern and making the one way call AddLine(...);
into a two way AddLine(...) -> ();, we can provide feedback to the client.
This will allow the client to throttle its output as appropriate. In this
example, we'll simply have the client wait for the ack before sending the next
message it has waiting, though more complex designs could send messages
optimistically, and only throttle when they receive async acks less frequently
than expected.
First, we need to define our interface definitions and test harness. The FIDL, CML, and realm interface definitions set up a scaffold that arbitrary implementations can use:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.canvas.addlinemetered; /// A point in 2D space. type Point = struct { x int64; y int64; }; /// A line in 2D space. alias Line = array<Point, 2>; /// A bounding box in 2D space. This is the result of "drawing" operations on our canvas, and what /// the server reports back to the client. These bounds are sufficient to contain all of the /// lines (inclusive) on a canvas at a given time. type BoundingBox = struct { top_left Point; bottom_right Point; }; /// Manages a single instance of a canvas. Each session of this protocol is responsible for a new /// canvas. @discoverable open protocol Instance { /// Add a line to the canvas. /// /// This method can be considered an improvement over the one-way case from a flow control /// perspective, as it is now much more difficult for a well-behaved client to "get ahead" of /// the server and overwhelm. This is because the client now waits for each request to be acked /// by the server before proceeding. This change represents a trade-off: we get much greater /// synchronization of message flow between the client and the server, at the cost of worse /// performance at the limit due to the extra wait imposed by each ack. flexible AddLine(struct { line Line; }) -> (); /// Update the client with the latest drawing state. The server makes no guarantees about how /// often this event occurs - it could occur multiple times per board state, for example. flexible -> OnDrawn(BoundingBox); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.canvas.addlinemetered.Instance" }, ], config: { // A script for the client to follow. Entries in the script may take one of two forms: a // pair of signed-integer coordinates like "-2,15:4,5", or the string "WAIT". The former // calls `AddLine(...)`, while the latter pauses execution until the next `->OnDrawn(...)` // event is received. // // TODO(https://fxbug.dev/42178362): It would absolve individual language implementations of a great // deal of string parsing if we were able to use a vector of `union { Point; WaitEnum}` // here. script: { type: "vector", max_count: 100, element: { type: "string", max_size: 64, }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.canvas.addlinemetered.Instance" }, ], expose: [ { protocol: "examples.canvas.addlinemetered.Instance", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.canvas.addlinemetered.Instance", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{format_err, Context as _, Error}; use config::Config; use fidl_examples_canvas_addlinemetered::{InstanceEvent, InstanceMarker, Point}; use fuchsia_component::client::connect_to_protocol; use futures::TryStreamExt; use std::{thread, time}; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send Instance requests // across the channel. let instance = connect_to_protocol::<InstanceMarker>()?; println!("Outgoing connection enabled"); for action in config.script.into_iter() { // If the next action in the script is to "WAIT", block until an OnDrawn event is received // from the server. if action == "WAIT" { let mut event_stream = instance.take_event_stream(); loop { match event_stream .try_next() .await .context("Error getting event response from proxy")? .ok_or_else(|| format_err!("Proxy sent no events"))? { InstanceEvent::OnDrawn { top_left, bottom_right } => { println!( "OnDrawn event received: top_left: {:?}, bottom_right: {:?}", top_left, bottom_right ); break; } InstanceEvent::_UnknownEvent { ordinal, .. } => { println!("Received an unknown event with ordinal {ordinal}"); } } } continue; } // If the action is not a "WAIT", we need to draw a line instead. Parse the string input, // making two points out of it. let mut points = action .split(":") .map(|point| { let integers = point .split(",") .map(|integer| integer.parse::<i64>().unwrap()) .collect::<Vec<i64>>(); Point { x: integers[0], y: integers[1] } }) .collect::<Vec<Point>>(); // Assemble a line from the two points. let from = points.pop().ok_or_else(|| format_err!("line requires 2 points, but has 0"))?; let to = points.pop().ok_or_else(|| format_err!("line requires 2 points, but has 1"))?; let line = [from, to]; // Draw a line to the canvas by calling the server, using the two points we just parsed // above as arguments. println!("AddLine request sent: {:?}", line); // By awaiting on the reply, we prevent the client from sending another request before the // server is ready to handle, thereby syncing the flow rate between the two parties over // this method. instance.add_line(&line).await.context("Error sending request")?; println!("AddLine response received"); } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fidl::endpoints::RequestStream as _; use fidl_examples_canvas_addlinemetered::{ BoundingBox, InstanceRequest, InstanceRequestStream, Point, }; use fuchsia_async::{MonotonicInstant, Timer}; use fuchsia_component::server::ServiceFs; use futures::future::join; use futures::prelude::*; use std::sync::{Arc, Mutex}; // A struct that stores the two things we care about for this example: the bounding box the lines // that have been added thus far, and bit to track whether or not there have been changes since the // last `OnDrawn` event. #[derive(Debug)] struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. changed: bool, bounding_box: BoundingBox, } impl CanvasState { /// Handler for the `AddLine` method. fn add_line(&mut self, line: [Point; 2]) { // Update the bounding box to account for the new lines we've just "added" to the canvas. let bounds = &mut self.bounding_box; for point in line { if point.x < bounds.top_left.x { bounds.top_left.x = point.x; } if point.y > bounds.top_left.y { bounds.top_left.y = point.y; } if point.x > bounds.bottom_right.x { bounds.bottom_right.x = point.x; } if point.y < bounds.bottom_right.y { bounds.bottom_right.y = point.y; } } // Mark the state as "dirty", so that an update is sent back to the client on the next tick. self.changed = true } } /// Creates a new instance of the server, paired to a single client across a zircon channel. async fn run_server(stream: InstanceRequestStream) -> Result<(), Error> { // Create a new in-memory state store for the state of the canvas. The store will live for the // lifetime of the connection between the server and this particular client. let state = Arc::new(Mutex::new(CanvasState { changed: true, bounding_box: BoundingBox { top_left: Point { x: 0, y: 0 }, bottom_right: Point { x: 0, y: 0 }, }, })); // Take ownership of the control_handle from the stream, which will allow us to push events from // a different async task. let control_handle = stream.control_handle(); // A separate watcher task periodically "draws" the canvas, and notifies the client of the new // state. We'll need a cloned reference to the canvas state to be accessible from the new // task. let state_ref = state.clone(); let update_sender = || async move { loop { // Our server sends one update per second. Timer::new(MonotonicInstant::after(zx::MonotonicDuration::from_seconds(1))).await; let mut state = state_ref.lock().unwrap(); if !state.changed { continue; } // After acquiring the lock, this is where we would draw the actual lines. Since this is // just an example, we'll avoid doing the actual rendering, and simply send the bounding // box to the client instead. let bounds = state.bounding_box; match control_handle.send_on_drawn(&bounds.top_left, &bounds.bottom_right) { Ok(_) => println!( "OnDrawn event sent: top_left: {:?}, bottom_right: {:?}", bounds.top_left, bounds.bottom_right ), Err(_) => return, } // Reset the change tracker. state.changed = false } }; // Handle requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. let state_ref = &state; let request_handler = stream.map(|result| result.context("failed request")).try_for_each(|request| async move { // Match based on the method being invoked. match request { InstanceRequest::AddLine { line, responder } => { println!("AddLine request received: {:?}", line); state_ref.lock().unwrap().add_line(line); // Because this is now a two-way method, we must use the generated `responder` // to send an in this case empty reply back to the client. This is the mechanic // which syncs the flow rate between the client and server on this method, // thereby preventing the client from "flooding" the server with unacknowledged // work. responder.send().context("Error responding")?; println!("AddLine response sent"); } // InstanceRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }); // This await does not complete, and thus the function does not return, unless the server errors // out. The stream will await indefinitely, thereby creating a long-lived server. Here, we first // wait for the updater task to realize the connection has died, then bubble up the error. join(request_handler, update_sender()).await.0 } // A helper enum that allows us to treat a `Instance` service instance as a value. enum IncomingService { Instance(InstanceRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Instance` protocol - this will allow the client to see // the server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Instance); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Instance(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.addlinemetered/cpp/fidl.h> #include <lib/async-loop/cpp/loop.h> #include <lib/component/incoming/cpp/protocol.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <charconv> #include <examples/fidl/new/canvas/add_line_metered/cpp_natural/client/config.h> // The |EventHandler| is a derived class that we pass into the |fidl::WireClient| to handle incoming // events asynchronously. class EventHandler : public fidl::AsyncEventHandler<examples_canvas_addlinemetered::Instance> { public: // Handler for |OnDrawn| events sent from the server. void OnDrawn(fidl::Event<examples_canvas_addlinemetered::Instance::OnDrawn>& event) override { auto top_left = event.top_left(); auto bottom_right = event.bottom_right(); FX_LOGS(INFO) << "OnDrawn event received: top_left: Point { x: " << top_left.x() << ", y: " << top_left.y() << " }, bottom_right: Point { x: " << bottom_right.x() << ", y: " << bottom_right.y() << " }"; loop_.Quit(); } void on_fidl_error(fidl::UnbindInfo error) override { FX_LOGS(ERROR) << error; } void handle_unknown_event( fidl::UnknownEventMetadata<examples_canvas_addlinemetered::Instance> metadata) override { FX_LOGS(WARNING) << "Received an unknown event with ordinal " << metadata.event_ordinal; } explicit EventHandler(async::Loop& loop) : loop_(loop) {} private: async::Loop& loop_; }; // A helper function that takes a coordinate in string form, like "123,-456", and parses it into a // a struct of the form |{ in64 x; int64 y; }|. ::examples_canvas_addlinemetered::Point ParsePoint(std::string_view input) { int64_t x = 0; int64_t y = 0; size_t index = input.find(','); if (index != std::string::npos) { std::from_chars(input.data(), input.data() + index, x); std::from_chars(input.data() + index + 1, input.data() + input.length(), y); } return ::examples_canvas_addlinemetered::Point(x, y); } // A helper function that takes a coordinate pair in string form, like "1,2:-3,-4", and parses it // into an array of 2 |Point| structs. ::std::array<::examples_canvas_addlinemetered::Point, 2> ParseLine(const std::string& action) { auto input = std::string_view(action); size_t index = input.find(':'); if (index != std::string::npos) { return {ParsePoint(input.substr(0, index)), ParsePoint(input.substr(index + 1))}; } return {}; } int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop and dispatcher. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace. This can fail so it's wrapped in a // |zx::result| and it must be checked for errors. zx::result client_end = component::Connect<examples_canvas_addlinemetered::Instance>(); if (!client_end.is_ok()) { FX_LOGS(ERROR) << "Synchronous error when connecting to the |Instance| protocol: " << client_end.status_string(); return -1; } // Create an instance of the event handler. EventHandler event_handler(loop); // Create an asynchronous client using the newly-established connection. fidl::Client client(std::move(*client_end), dispatcher, &event_handler); FX_LOGS(INFO) << "Outgoing connection enabled"; for (const auto& action : conf.script()) { // If the next action in the script is to "WAIT", block until an |OnDrawn| event is received // from the server. if (action == "WAIT") { loop.Run(); loop.ResetQuit(); continue; } // Draw a line to the canvas by calling the server, using the two points we just parsed // above as arguments. auto line = ParseLine(action); FX_LOGS(INFO) << "AddLine request sent: [Point { x: " << line[1].x() << ", y: " << line[1].y() << " }, Point { x: " << line[0].x() << ", y: " << line[0].y() << " }]"; client->AddLine(line).ThenExactlyOnce( [&](fidl::Result<examples_canvas_addlinemetered::Instance::AddLine>& result) { // Check if the FIDL call succeeded or not. if (!result.is_ok()) { // Check that our two-way call succeeded, and handle the error appropriately. In the // case of this example, there is nothing we can do to recover here, except to log an // error and exit the program. FX_LOGS(ERROR) << "Could not send AddLine request: " << result.error_value().FormatDescription(); } FX_LOGS(INFO) << "AddLine response received"; // Quit the loop, thereby handing control back to the outer loop of actions being iterated // over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.addlinemetered/cpp/fidl.h> #include <lib/async-loop/cpp/loop.h> #include <lib/async/cpp/task.h> #include <lib/component/outgoing/cpp/outgoing_directory.h> #include <lib/fidl/cpp/wire/channel.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <src/lib/fxl/macros.h> #include <src/lib/fxl/memory/weak_ptr.h> // A struct that stores the two things we care about for this example: the set of lines, and the // bounding box that contains them. struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. bool changed = true; examples_canvas_addlinemetered::BoundingBox bounding_box; }; // An implementation of the |Instance| protocol. class InstanceImpl final : public fidl::Server<examples_canvas_addlinemetered::Instance> { public: // Bind this implementation to a channel. InstanceImpl(async_dispatcher_t* dispatcher, fidl::ServerEnd<examples_canvas_addlinemetered::Instance> server_end) : binding_(fidl::BindServer( dispatcher, std::move(server_end), this, [this](InstanceImpl* impl, fidl::UnbindInfo info, fidl::ServerEnd<examples_canvas_addlinemetered::Instance> server_end) { if (info.reason() != ::fidl::Reason::kPeerClosedWhileReading) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; })), weak_factory_(this) { // Start the update timer on startup. Our server sends one update per second ScheduleOnDrawnEvent(dispatcher, zx::sec(1)); } void AddLine(AddLineRequest& request, AddLineCompleter::Sync& completer) override { auto points = request.line(); FX_LOGS(INFO) << "AddLine request received: [Point { x: " << points[1].x() << ", y: " << points[1].y() << " }, Point { x: " << points[0].x() << ", y: " << points[0].y() << " }]"; // Update the bounding box to account for the new line we've just "added" to the canvas. auto& bounds = state_.bounding_box; for (const auto& point : request.line()) { if (point.x() < bounds.top_left().x()) { bounds.top_left().x() = point.x(); } if (point.y() > bounds.top_left().y()) { bounds.top_left().y() = point.y(); } if (point.x() > bounds.bottom_right().x()) { bounds.bottom_right().x() = point.x(); } if (point.y() < bounds.bottom_right().y()) { bounds.bottom_right().y() = point.y(); } } // Mark the state as "dirty", so that an update is sent back to the client on the next |OnDrawn| // event. state_.changed = true; // Because this is now a two-way method, we must use the generated |completer| to send an in // this case empty reply back to the client. This is the mechanic which syncs the flow rate // between the client and server on this method, thereby preventing the client from "flooding" // the server with unacknowledged work. completer.Reply(); FX_LOGS(INFO) << "AddLine response sent"; } void handle_unknown_method( fidl::UnknownMethodMetadata<examples_canvas_addlinemetered::Instance> metadata, fidl::UnknownMethodCompleter::Sync& completer) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << metadata.method_ordinal; } private: // Each scheduled update waits for the allotted amount of time, sends an update if something has // changed, and schedules the next update. void ScheduleOnDrawnEvent(async_dispatcher_t* dispatcher, zx::duration after) { async::PostDelayedTask( dispatcher, [&, dispatcher, after, weak = weak_factory_.GetWeakPtr()] { // Halt execution if the binding has been deallocated already. if (!weak) { return; } // Schedule the next update if the binding still exists. weak->ScheduleOnDrawnEvent(dispatcher, after); // No need to send an update if nothing has changed since the last one. if (!weak->state_.changed) { return; } // This is where we would draw the actual lines. Since this is just an example, we'll // avoid doing the actual rendering, and simply send the bounding box to the client // instead. auto result = fidl::SendEvent(binding_)->OnDrawn(state_.bounding_box); if (!result.is_ok()) { return; } auto top_left = state_.bounding_box.top_left(); auto bottom_right = state_.bounding_box.bottom_right(); FX_LOGS(INFO) << "OnDrawn event sent: top_left: Point { x: " << top_left.x() << ", y: " << top_left.y() << " }, bottom_right: Point { x: " << bottom_right.x() << ", y: " << bottom_right.y() << " }"; // Reset the change tracker. state_.changed = false; }, after); } fidl::ServerBindingRef<examples_canvas_addlinemetered::Instance> binding_; CanvasState state_ = CanvasState{}; // Generates weak references to this object, which are appropriate to pass into asynchronous // callbacks that need to access this object. The references are automatically invalidated // if this object is destroyed. fxl::WeakPtrFactory<InstanceImpl> weak_factory_; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. This // directory is where the outgoing FIDL protocols are installed so that they can be provided to // other components. component::OutgoingDirectory outgoing = component::OutgoingDirectory(dispatcher); // The `ServeFromStartupInfo()` function sets up the outgoing directory with the startup handle. // The startup handle is a handle provided to every component by the system, so that they can // serve capabilities (e.g. FIDL protocols) to other components. zx::result result = outgoing.ServeFromStartupInfo(); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to serve outgoing directory: " << result.status_string(); return -1; } // Register a handler for components trying to connect to // |examples.canvas.addlinemetered.Instance|. result = outgoing.AddUnmanagedProtocol<examples_canvas_addlinemetered::Instance>( [dispatcher](fidl::ServerEnd<examples_canvas_addlinemetered::Instance> server_end) { // Create an instance of our InstanceImpl that destroys itself when the connection closes. new InstanceImpl(dispatcher, std::move(server_end)); }); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to add Instance protocol: " << result.status_string(); return -1; } // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
C++ (Wire)
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.addlinemetered/cpp/wire.h> #include <lib/async-loop/cpp/loop.h> #include <lib/component/incoming/cpp/protocol.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <charconv> #include <examples/fidl/new/canvas/add_line_metered/cpp_wire/client/config.h> // The |EventHandler| is a derived class that we pass into the |fidl::WireClient| to handle incoming // events asynchronously. class EventHandler : public fidl::WireAsyncEventHandler<examples_canvas_addlinemetered::Instance> { public: // Handler for |OnDrawn| events sent from the server. void OnDrawn(fidl::WireEvent<examples_canvas_addlinemetered::Instance::OnDrawn>* event) override { auto top_left = event->top_left; auto bottom_right = event->bottom_right; FX_LOGS(INFO) << "OnDrawn event received: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; loop_.Quit(); } void on_fidl_error(fidl::UnbindInfo error) override { FX_LOGS(ERROR) << error; } void handle_unknown_event( fidl::UnknownEventMetadata<examples_canvas_addlinemetered::Instance> metadata) override { FX_LOGS(WARNING) << "Received an unknown event with ordinal " << metadata.event_ordinal; } explicit EventHandler(async::Loop& loop) : loop_(loop) {} private: async::Loop& loop_; }; // A helper function that takes a coordinate in string form, like "123,-456", and parses it into a // a struct of the form |{ in64 x; int64 y; }|. ::examples_canvas_addlinemetered::wire::Point ParsePoint(std::string_view input) { int64_t x = 0; int64_t y = 0; size_t index = input.find(','); if (index != std::string::npos) { std::from_chars(input.data(), input.data() + index, x); std::from_chars(input.data() + index + 1, input.data() + input.length(), y); } return ::examples_canvas_addlinemetered::wire::Point{.x = x, .y = y}; } // A helper function that takes a coordinate pair in string form, like "1,2:-3,-4", and parses it // into an array of 2 |Point| structs. ::fidl::Array<::examples_canvas_addlinemetered::wire::Point, 2> ParseLine( const std::string& action) { auto input = std::string_view(action); size_t index = input.find(':'); if (index != std::string::npos) { return {ParsePoint(input.substr(0, index)), ParsePoint(input.substr(index + 1))}; } return {}; } int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop and dispatcher. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace. This can fail so it's wrapped in a // |zx::result| and it must be checked for errors. zx::result client_end = component::Connect<examples_canvas_addlinemetered::Instance>(); if (!client_end.is_ok()) { FX_LOGS(ERROR) << "Synchronous error when connecting to the |Instance| protocol: " << client_end.status_string(); return -1; } // Create an instance of the event handler. EventHandler event_handler(loop); // Create an asynchronous client using the newly-established connection. fidl::WireClient client(std::move(*client_end), dispatcher, &event_handler); FX_LOGS(INFO) << "Outgoing connection enabled"; for (const auto& action : conf.script()) { // If the next action in the script is to "WAIT", block until an |OnDrawn| event is received // from the server. if (action == "WAIT") { loop.Run(); loop.ResetQuit(); continue; } // Draw a line to the canvas by calling the server, using the two points we just parsed // above as arguments. auto line = ParseLine(action); FX_LOGS(INFO) << "AddLine request sent: [Point { x: " << line[1].x << ", y: " << line[1].y << " }, Point { x: " << line[0].x << ", y: " << line[0].y << " }]"; client->AddLine(line).ThenExactlyOnce( [&](fidl::WireUnownedResult<examples_canvas_addlinemetered::Instance::AddLine>& result) { // Check if the FIDL call succeeded or not. if (!result.ok()) { // Check that our two-way call succeeded, and handle the error appropriately. In the // case of this example, there is nothing we can do to recover here, except to log an // error and exit the program. FX_LOGS(ERROR) << "Could not send AddLine request: " << result.status_string(); } FX_LOGS(INFO) << "AddLine response received"; // Quit the loop, thereby handing control back to the outer loop of actions being iterated // over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.addlinemetered/cpp/wire.h> #include <lib/async-loop/cpp/loop.h> #include <lib/async/cpp/task.h> #include <lib/component/outgoing/cpp/outgoing_directory.h> #include <lib/fidl/cpp/wire/channel.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <src/lib/fxl/macros.h> #include <src/lib/fxl/memory/weak_ptr.h> // A struct that stores the two things we care about for this example: the set of lines, and the // bounding box that contains them. struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. bool changed = true; examples_canvas_addlinemetered::wire::BoundingBox bounding_box; }; // An implementation of the |Instance| protocol. class InstanceImpl final : public fidl::WireServer<examples_canvas_addlinemetered::Instance> { public: // Bind this implementation to a channel. InstanceImpl(async_dispatcher_t* dispatcher, fidl::ServerEnd<examples_canvas_addlinemetered::Instance> server_end) : binding_(fidl::BindServer( dispatcher, std::move(server_end), this, [this](InstanceImpl* impl, fidl::UnbindInfo info, fidl::ServerEnd<examples_canvas_addlinemetered::Instance> server_end) { if (info.reason() != ::fidl::Reason::kPeerClosedWhileReading) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; })), weak_factory_(this) { // Start the update timer on startup. Our server sends one update per second ScheduleOnDrawnEvent(dispatcher, zx::sec(1)); } void AddLine(AddLineRequestView request, AddLineCompleter::Sync& completer) override { auto points = request->line; FX_LOGS(INFO) << "AddLine request received: [Point { x: " << points[1].x << ", y: " << points[1].y << " }, Point { x: " << points[0].x << ", y: " << points[0].y << " }]"; // Update the bounding box to account for the new line we've just "added" to the canvas. auto& bounds = state_.bounding_box; for (const auto& point : request->line) { if (point.x < bounds.top_left.x) { bounds.top_left.x = point.x; } if (point.y > bounds.top_left.y) { bounds.top_left.y = point.y; } if (point.x > bounds.bottom_right.x) { bounds.bottom_right.x = point.x; } if (point.y < bounds.bottom_right.y) { bounds.bottom_right.y = point.y; } } // Mark the state as "dirty", so that an update is sent back to the client on the next |OnDrawn| // event. state_.changed = true; // Because this is now a two-way method, we must use the generated |completer| to send an in // this case empty reply back to the client. This is the mechanic which syncs the flow rate // between the client and server on this method, thereby preventing the client from "flooding" // the server with unacknowledged work. completer.Reply(); FX_LOGS(INFO) << "AddLine response sent"; } void handle_unknown_method( fidl::UnknownMethodMetadata<examples_canvas_addlinemetered::Instance> metadata, fidl::UnknownMethodCompleter::Sync& completer) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << metadata.method_ordinal; } private: // Each scheduled update waits for the allotted amount of time, sends an update if something has // changed, and schedules the next update. void ScheduleOnDrawnEvent(async_dispatcher_t* dispatcher, zx::duration after) { async::PostDelayedTask( dispatcher, [&, dispatcher, after, weak = weak_factory_.GetWeakPtr()] { // Halt execution if the binding has been deallocated already. if (!weak) { return; } // Schedule the next update if the binding still exists. weak->ScheduleOnDrawnEvent(dispatcher, after); // No need to send an update if nothing has changed since the last one. if (!weak->state_.changed) { return; } // This is where we would draw the actual lines. Since this is just an example, we'll // avoid doing the actual rendering, and simply send the bounding box to the client // instead. auto top_left = weak->state_.bounding_box.top_left; auto bottom_right = weak->state_.bounding_box.bottom_right; fidl::Status status = fidl::WireSendEvent(weak->binding_)->OnDrawn(top_left, bottom_right); if (!status.ok()) { return; } FX_LOGS(INFO) << "OnDrawn event sent: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; // Reset the change tracker. weak->state_.changed = false; }, after); } fidl::ServerBindingRef<examples_canvas_addlinemetered::Instance> binding_; CanvasState state_ = CanvasState{}; // Generates weak references to this object, which are appropriate to pass into asynchronous // callbacks that need to access this object. The references are automatically invalidated // if this object is destroyed. fxl::WeakPtrFactory<InstanceImpl> weak_factory_; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. This // directory is where the outgoing FIDL protocols are installed so that they can be provided to // other components. component::OutgoingDirectory outgoing = component::OutgoingDirectory(dispatcher); // The `ServeFromStartupInfo()` function sets up the outgoing directory with the startup handle. // The startup handle is a handle provided to every component by the system, so that they can // serve capabilities (e.g. FIDL protocols) to other components. zx::result result = outgoing.ServeFromStartupInfo(); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to serve outgoing directory: " << result.status_string(); return -1; } // Register a handler for components trying to connect to // |examples.canvas.addlinemetered.Instance|. result = outgoing.AddUnmanagedProtocol<examples_canvas_addlinemetered::Instance>( [dispatcher](fidl::ServerEnd<examples_canvas_addlinemetered::Instance> server_end) { // Create an instance of our InstanceImpl that destroys itself when the connection closes. new InstanceImpl(dispatcher, std::move(server_end)); }); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to add Instance protocol: " << result.status_string(); return -1; } // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
HLCPP
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <lib/async-loop/cpp/loop.h> #include <lib/sys/cpp/component_context.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <charconv> #include <examples/canvas/addlinemetered/cpp/fidl.h> #include <examples/fidl/new/canvas/add_line_metered/hlcpp/client/config.h> #include "lib/fpromise/result.h" // A helper function that takes a coordinate in string form, like "123,-456", and parses it into a // a struct of the form |{ in64 x; int64 y; }|. ::examples::canvas::addlinemetered::Point ParsePoint(std::string_view input) { int64_t x = 0; int64_t y = 0; size_t index = input.find(','); if (index != std::string::npos) { std::from_chars(input.data(), input.data() + index, x); std::from_chars(input.data() + index + 1, input.data() + input.length(), y); } return ::examples::canvas::addlinemetered::Point{.x = x, .y = y}; } // A helper function that takes a coordinate pair in string form, like "1,2:-3,-4", and parses it // into an array of 2 |Point| structs. ::std::array<::examples::canvas::addlinemetered::Point, 2> ParseLine(const std::string& action) { auto input = std::string_view(action); size_t index = input.find(':'); if (index != std::string::npos) { return {ParsePoint(input.substr(0, index)), ParsePoint(input.substr(index + 1))}; } return {}; } int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace, then create an asynchronous client // using the newly-established connection. examples::canvas::addlinemetered::InstancePtr instance_proxy; auto context = sys::ComponentContext::Create(); context->svc()->Connect(instance_proxy.NewRequest(dispatcher)); FX_LOGS(INFO) << "Outgoing connection enabled"; instance_proxy.set_error_handler([&loop](zx_status_t status) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; loop.Quit(); }); // Provide a lambda to handle incoming |OnDrawn| events asynchronously. instance_proxy.events().OnDrawn = [&loop]( ::examples::canvas::addlinemetered::Point top_left, ::examples::canvas::addlinemetered::Point bottom_right) { FX_LOGS(INFO) << "OnDrawn event received: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; loop.Quit(); }; instance_proxy.events().handle_unknown_event = [](uint64_t ordinal) { FX_LOGS(WARNING) << "Received an unknown event with ordinal " << ordinal; }; for (const auto& action : conf.script()) { // If the next action in the script is to "WAIT", block until an |OnDrawn| event is received // from the server. if (action == "WAIT") { loop.Run(); loop.ResetQuit(); continue; } // Draw a line to the canvas by calling the server, using the two points we just parsed // above as arguments. auto line = ParseLine(action); FX_LOGS(INFO) << "AddLine request sent: [Point { x: " << line[1].x << ", y: " << line[1].y << " }, Point { x: " << line[0].x << ", y: " << line[0].y << " }]"; instance_proxy->AddLine(line, [&](fpromise::result<void, fidl::FrameworkErr> result) { if (result.is_error()) { // Check that our flexible two-way call was known to the server and handle the case of an // unknown method appropriately. In the case of this example, there is nothing we can do to // recover here, except to log an error and exit the program. FX_LOGS(ERROR) << "Server does not implement AddLine"; } FX_LOGS(INFO) << "AddLine response received"; // Quit the loop, thereby handing control back to the outer loop of actions being iterated // over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <lib/async-loop/cpp/loop.h> #include <lib/async-loop/default.h> #include <lib/async/cpp/task.h> #include <lib/fidl/cpp/binding.h> #include <lib/sys/cpp/component_context.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <examples/canvas/addlinemetered/cpp/fidl.h> #include <src/lib/fxl/macros.h> #include <src/lib/fxl/memory/weak_ptr.h> // A struct that stores the two things we care about for this example: the set of lines, and the // bounding box that contains them. struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. bool changed = true; examples::canvas::addlinemetered::BoundingBox bounding_box; }; // An implementation of the |Instance| protocol. class InstanceImpl final : public examples::canvas::addlinemetered::Instance { public: // Bind this implementation to an |InterfaceRequest|. InstanceImpl(async_dispatcher_t* dispatcher, fidl::InterfaceRequest<examples::canvas::addlinemetered::Instance> request) : binding_(fidl::Binding<examples::canvas::addlinemetered::Instance>(this)), weak_factory_(this) { binding_.Bind(std::move(request), dispatcher); // Gracefully handle abrupt shutdowns. binding_.set_error_handler([this](zx_status_t status) mutable { if (status != ZX_ERR_PEER_CLOSED) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; }); // Start the update timer on startup. Our server sends one update per second. ScheduleOnDrawnEvent(dispatcher, zx::sec(1)); } void AddLine(::std::array<::examples::canvas::addlinemetered::Point, 2> line, AddLineCallback callback) override { FX_LOGS(INFO) << "AddLine request received: [Point { x: " << line[1].x << ", y: " << line[1].y << " }, Point { x: " << line[0].x << ", y: " << line[0].y << " }]"; // Update the bounding box to account for the new line we've just "added" to the canvas. auto& bounds = state_.bounding_box; for (const auto& point : line) { if (point.x < bounds.top_left.x) { bounds.top_left.x = point.x; } if (point.y > bounds.top_left.y) { bounds.top_left.y = point.y; } if (point.x > bounds.bottom_right.x) { bounds.bottom_right.x = point.x; } if (point.y < bounds.bottom_right.y) { bounds.bottom_right.y = point.y; } } // Mark the state as "dirty", so that an update is sent back to the client on the next |OnDrawn| // event. state_.changed = true; // Because this is now a two-way method, we must use the generated |callback| to send an in // this case empty reply back to the client. This is the mechanic which syncs the flow rate // between the client and server on this method, thereby preventing the client from "flooding" // the server with unacknowledged work. callback(fpromise::ok()); FX_LOGS(INFO) << "AddLine response sent"; } void handle_unknown_method(uint64_t ordinal, bool method_has_response) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << ordinal; } private: // Each scheduled update waits for the allotted amount of time, sends an update if something has // changed, and schedules the next update. void ScheduleOnDrawnEvent(async_dispatcher_t* dispatcher, zx::duration after) { async::PostDelayedTask( dispatcher, [&, dispatcher, after, weak = weak_factory_.GetWeakPtr()] { // Halt execution if the binding has been deallocated already. if (!weak) { return; } // Schedule the next update if the binding still exists. weak->ScheduleOnDrawnEvent(dispatcher, after); // No need to send an update if nothing has changed since the last one. if (!weak->state_.changed) { return; } // This is where we would draw the actual lines. Since this is just an example, we'll // avoid doing the actual rendering, and simply send the bounding box to the client // instead. auto top_left = state_.bounding_box.top_left; auto bottom_right = state_.bounding_box.bottom_right; binding_.events().OnDrawn(top_left, bottom_right); FX_LOGS(INFO) << "OnDrawn event sent: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; // Reset the change tracker. state_.changed = false; }, after); } fidl::Binding<examples::canvas::addlinemetered::Instance> binding_; CanvasState state_ = CanvasState{}; // Generates weak references to this object, which are appropriate to pass into asynchronous // callbacks that need to access this object. The references are automatically invalidated // if this object is destroyed. fxl::WeakPtrFactory<InstanceImpl> weak_factory_; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. // // Note that unlike the new C++ bindings, HLCPP bindings rely on the async loop being attached to // the current thread via the |kAsyncLoopConfigAttachToCurrentThread| configuration. async::Loop loop(&kAsyncLoopConfigAttachToCurrentThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. // This directory is where the outgoing FIDL protocols are installed so that they can be // provided to other components. auto context = sys::ComponentContext::CreateAndServeOutgoingDirectory(); // Register a handler for components trying to connect to // |examples.canvas.addlinemetered.Instance|. context->outgoing()->AddPublicService( fidl::InterfaceRequestHandler<examples::canvas::addlinemetered::Instance>( [dispatcher](fidl::InterfaceRequest<examples::canvas::addlinemetered::Instance> request) { // Create an instance of our |InstanceImpl| that destroys itself when the connection // closes. new InstanceImpl(dispatcher, std::move(request)); })); // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
Alias
FIDL recipe: Alias
An alias is a FIDL declaration that assigns a new name to an existing type.
This has several benefits:
- Using
aliasensures that there is a single source of truth for the concept the aliased type represents. - It provides a way to name things, especially constrained types.
- Disparate uses of the now-aliased type may be linked as being instances of the same concept.
It is important to note that aliases do not carry through to the generated
bindings code at the moment. In other words, the name assigned to an alias
declaration will never appear as a declaration name in the generated FIDL code.
In this example, adding an alias for Key allows us to avoid repetition with
a bespoke name, while also making clear to the reader that both the key value
on the Item type and the key used in the ReadItem request struct are
purposefully, and not merely coincidentally, the same thing.
Reasoning
The original write-only key-value store is now extended with the ability to read items back out of the store.
Implementation
The changes applied to the FIDL and CML definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.addreaditem; // Aliases for the key and value. Using aliases helps increase the readability of FIDL files and // reduces likelihood of errors due to differing constraints. alias Key = string:128; alias Value = vector<byte>:64000; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key Key; value Value; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// An enumeration of things that may go wrong when trying to read a value out of our store. type ReadError = flexible enum { UNKNOWN = 0; NOT_FOUND = 1; }; /// A very basic key-value store - so basic, in fact, that one may only write to it, never read! @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; /// Reads an item from the store. flexible ReadItem(struct { key Key; }) -> (Item) error ReadError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.addreaditem.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, read_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.addreaditem.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.addreaditem.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.addreaditem.Store", from: "#server", to: "#client", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations for all languages change as well:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use config::Config; use fidl_examples_keyvaluestore_addreaditem::{Item, StoreMarker}; use fuchsia_component::client::connect_to_protocol; use std::{str, thread, time}; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; match store.write_item(&Item { key: key, value: value.into_bytes() }).await? { Ok(_) => println!("WriteItem Success"), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // The structured config for this client contains `read_items`, a vector of strings, each of // which is meant to be read from the key-value store. We iterate over these keys, attempting to // read them in turn. for key in config.read_items.into_iter() { let res = store.read_item(key.as_str()).await; match res.unwrap() { Ok(val) => { println!("ReadItem Success: key: {}, value: {}", key, str::from_utf8(&val.1)?) } Err(err) => println!("ReadItem Error: {}", err.into_primitive()), } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, fidl_examples_keyvaluestore_addreaditem::{ Item, ReadError, StoreRequest, StoreRequestStream, WriteError, }, fuchsia_component::server::ServiceFs, futures::prelude::*, lazy_static::lazy_static, regex::Regex, std::cell::RefCell, std::collections::hash_map::Entry, std::collections::HashMap, }; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z][A-Za-z0-9_\./]{2,62}[A-Za-z0-9]$") .expect("Key validation regex failed to compile"); } /// Handler for the `WriteItem` method. fn write_item(store: &mut HashMap<String, Vec<u8>>, attempt: Item) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Validate the value. if attempt.value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", attempt.key); return Err(WriteError::InvalidValue); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote value at key: {}", entry.key()); entry.insert(attempt.value); Ok(()) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, Vec<u8>>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.borrow_mut(), attempt)) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::ReadItem { key, responder } => { println!("ReadItem request received"); // Read the item from the store, returning the appropriate error if it could not be found. responder .send(match store.borrow().get(&key) { Some(found) => { println!("Read value at key: {}", key); Ok((&key, found)) } None => { println!("Read error: NOT_FOUND, For key: {}", key); Err(ReadError::NotFound) } }) .context("error sending reply")?; println!("ReadItem response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Anonymous type
FIDL recipe: Anonymous type
An anonymous type is a type whose definition is located inline with its use,
rather than in a standalone, named type declaration. There are two benefits to
using anonymous types. First, they prevent excessive namespace pollution,
absolving FIDL authors of the need to name types that are only used once.
Second, they prevent the type from being imported into another FIDL library via
the using declaration, as the type cannot be identified by name.
In this variant, we allow our key-value store to take other key-value stores as
members. In short, we turn it into a tree. We do this by replacing the original
definition of value with one that utilizes a two-member union: one variant
stores leaf nodes using the same vector<byte> type as before, while the other
stores branch nodes in the form of other nested stores.
Reasoning
Here, we see several uses of optionality, whereby we can declare a type that may or may not exist. There are three flavors of optionality in FIDL:
- Types that have are always stored
out-of-line
on the wire, and thus have a builtin way to describe "absentness" via the
null envelope. Enabling
optionality for these types doesn't affect the wire shape of messages they are
included in - it simply changes which values are valid for that particular
type. The
union,vector<T>,client_end,server_end, andzx.Handletypes can all be made optional via the addition of the:optionalconstraint. By making ourvalueunionoptional, we are able to introduce a canonical "null" entry, in the form of an absentvalue. This means that emptybytesand absent/emptystoreproperties are invalid values. - Unlike the aforementioned types, the
structlayout has no extra space where a null header can be stored. Because of this, it needs to be wrapped in an envelope, changing the on-the-wire shape of the message it is being included in. To ensure that this wire-modifying effect easily legible, theItemstructtype must be wrapped in abox<T>type template. - Finally,
tablelayouts are always optional. An absenttableis simply one with none of its members set.
Trees are a naturally self-referential data structure: any node in the tree may
contain a leaf with pure data (in this case, a string), or a sub-tree with more
nodes. This requires recursion: the definition of Item is now transitively
dependent on itself! Representing recursive types in FIDL can be a bit tricky,
especially because support is currently somewhat
limited. We can support such types as long as there is
at least one optional type in the cycle created by the self-reference. For
instance, here we define the items struct member to be a box<Item>,
thereby breaking the includes cycle.
These changes also make heavy use of anonymous types, or types whose
declarations are inlined at their sole point of use, rather than being named,
top-level type declarations of their own. By default, the names of anonymous
types in the generated language bindings are taken from their local context. For
instance, the newly introduced flexible union takes on its owning member's
name Value, the newly introduced struct would become Store, and so on.
Because this heuristic can sometimes cause collisions, FIDL provides an escape
hatch by allowing the author to manually override an anonymous type's generated
name. This is done via the @generated_name attribute, which allows one to
change the name generated by backends. We can use one here, where the would-be
Store type is renamed to NestedStore to prevent a name collision with the
protocol declaration that uses that same name.
Implementation
The FIDL, CML, and realm interface definitions are modified as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.supporttrees; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value strict union { // Keep the original `bytes` as one of the options in the new union. All leaf nodes in the // tree must be `bytes`, or absent unions (representing empty). Empty byte arrays are // disallowed. 1: bytes vector<byte>:64000; // Allows a store within a store, thereby turning our flat key-value store into a tree // thereof. Note the use of `@generated_name` to prevent a type-name collision with the // `Store` protocol below, and the use of `box<T>` to ensure that there is a break in the // chain of recursion, thereby allowing `Item` to include itself in its own definition. // // This is a table so that added fields, like for example a `hash`, can be easily added in // the future. 2: store @generated_name("nested_store") table { 1: items vector<box<Item>>; }; }:optional; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// A very basic key-value store. @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.supporttrees.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A newline separated list nested entries. The first line should be the key // for the nested store, and each subsequent entry should be a pointer to a text file // containing the string value. The name of that text file (without the `.txt` suffix) will // serve as the entries key. write_nested: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A list of keys, all of which will be populated as null entries. write_null: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.supporttrees.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.supporttrees.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.supporttrees.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, config::Config, fidl_examples_keyvaluestore_supporttrees::{Item, NestedStore, StoreMarker, Value}, fuchsia_component::client::connect_to_protocol, std::{thread, time}, }; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; let res = store .write_item(&Item { key: key.clone(), value: Some(Box::new(Value::Bytes(value.into_bytes()))), }) .await; match res? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // Add nested entries to the key-value store as well. The entries are strings, where the first // line is the key of the entry, and each subsequent entry should be a pointer to a text file // containing the string value. The name of that text file (without the `.txt` suffix) will // serve as the entries key. for spec in config.write_nested.into_iter() { let mut items = vec![]; let mut nested_store = NestedStore::default(); let mut lines = spec.split("\n"); let key = lines.next().unwrap(); // For each entry, make a new entry in the `NestedStore` being built. for entry in lines { let path = format!("/pkg/data/{}.txt", entry); let contents = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; items.push(Some(Box::new(Item { key: entry.to_string(), value: Some(Box::new(Value::Bytes(contents.into()))), }))); } nested_store.items = Some(items); // Send the `NestedStore`, represented as a vector of values. let res = store .write_item(&Item { key: key.to_string(), value: Some(Box::new(Value::Store(nested_store))), }) .await; match res? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // Each entry in this list is a null value in the store. for key in config.write_null.into_iter() { match store.write_item(&Item { key: key.to_string(), value: None }).await? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. // Note: For the clarity of this example, allow code to be unused. #![allow(dead_code)] use { anyhow::{Context as _, Error}, fidl_examples_keyvaluestore_supporttrees::{ Item, StoreRequest, StoreRequestStream, Value, WriteError, }, fuchsia_component::server::ServiceFs, futures::prelude::*, lazy_static::lazy_static, regex::Regex, std::cell::RefCell, std::collections::hash_map::Entry, std::collections::HashMap, std::str::from_utf8, }; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } // A representation of a key-value store that can contain an arbitrarily deep nesting of other // key-value stores. #[allow(clippy::box_collection, reason = "mass allow for https://fxbug.dev/381896734")] enum StoreNode { Leaf(Option<Vec<u8>>), Branch(Box<HashMap<String, StoreNode>>), } /// Recursive item writer, which takes a `StoreNode` that may not necessarily be the root node, and /// writes an entry to it. fn write_item( store: &mut HashMap<String, StoreNode>, attempt: Item, path: &str, ) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { let key = format!("{}{}", &path, entry.key()); match attempt.value { // Null entries are allowed. None => { println!("Wrote value: NONE at key: {}", key); entry.insert(StoreNode::Leaf(None)); } Some(value) => match *value { // If this is a nested store, recursively make a new store to insert at this // position. Value::Store(entry_list) => { // Validate the value - absent stores, items lists with no children, or any // of the elements within that list being empty boxes, are all not allowed. if entry_list.items.is_some() { let items = entry_list.items.unwrap(); if !items.is_empty() && items.iter().all(|i| i.is_some()) { let nested_path = format!("{}/", key); let mut nested_store = HashMap::<String, StoreNode>::new(); for item in items.into_iter() { write_item(&mut nested_store, *item.unwrap(), &nested_path)?; } println!("Created branch at key: {}", key); entry.insert(StoreNode::Branch(Box::new(nested_store))); return Ok(()); } } println!("Write error: INVALID_VALUE, For key: {}", key); return Err(WriteError::InvalidValue); } // This is a simple leaf node on this branch. Value::Bytes(value) => { // Validate the value. if value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", key); return Err(WriteError::InvalidValue); } println!("Wrote key: {}, value: {:?}", key, from_utf8(&value).unwrap()); entry.insert(StoreNode::Leaf(Some(value))); } }, } Ok(()) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, StoreNode>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.borrow_mut(), attempt, "")) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Bits
FIDL recipe: Bits
The bits type is FIDL's way of representing the bit
array. It is used in cases where a
set of boolean flags is desired. The bits array is generally used "over" an
underlying subtype, which controls its bitwidth on the wire.
Reasoning
The key-value store baseline example's implementation was a good starting point, but one major drawback is that data is stored as raw bytes. FIDL is a richly typed language. Forcing data that is for instance a UTF-8 string to be stored as an untyped byte array erases this valuable type information for readers of the *.fidl file, as well as for programmers using bindings generated from it.
Implementation
The main goal of this change is to replace the baseline case's vector<byte>
typed value member with a union that stores many possible types. In fact, as
of this change a good survey of FIDL's
value types is on
offer:
- All of FIDL's builtin scalar types are used as variants in the
Valueunion:bool,uint8,uint16,uint32,uint64,int8,int16,int32,int64,float32, andfloat64(also known as FIDL's primitive types), as well asstring. - This
unionalso features uses of FIDL's builtinarray<T, N>andvector<T>type templates. - All of FIDL's type layouts, namely
bits,enum,table,union, andstruct, are utilized in this example at least once.
The request and response payloads used for WriteItem have also been changed
from structs to a named table and an inlined flexible union, respectively.
In fact, any of these three layouts may be used a request/response payload. The
latter two, known as table payloads and *union payloads, respectively, are
preferred in all but the most message size sensitive cases. This is because they
are much easier to extend in the future in a binary compatible way.
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.usegenericvalues; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value Value; }; // Because the `Value` must be used both in the request and the response, we give it its own named // type. The type is a `union` of all possible data types that we take as values, and is marked // `flexible` to allow for the easy addition of new data types in the future. type Value = flexible union { // Keep the original `bytes` as one of the options in the new union. 1: bytes vector<byte>:64000; // A `string` is very similar to `vector<byte>` on the wire, with the extra constraint that // it enforces that it enforces that the byte vector in question is valid UTF-8. 2: string string:64000; // All of FIDL's primitive types. 3: bool bool; 4: uint8 uint8; 5: int8 int8; 6: uint16 uint16; 7: int16 int16; 8: uint32 uint32; 9: int32 int32; 10: float32 float32; 11: uint64 uint64; 12: int64 int64; 13: float64 float64; // FIDL does not natively support 128-bit integer types, so we have to define our own // representations. 14: uint128 array<uint64, 2>; }; // Because we now supoprt a richer range of types as values in our store, it is helpful to use a // `flexible`, and therefore evolvable, `bits` type to store write options. type WriteOptions = flexible bits : uint8 { // This flag allows us to overwrite existing data when there is a collision, rather than failing // with an `WriteError.ALREADY_EXISTS`. OVERWRITE = 0b1; // This flag allows us to concatenate to existing data when there is a collision, rather than // failing with an `WriteError.ALREADY_EXISTS`. "Concatenation" means addition for the numeric // variants and appending to the `bytes`/`string` variants. If no existing data can be found, we // "concatenate" to default values of zero and an empty vector, respectively. Attempting to // concatenate to an existing variant of a different type will return a // `WriteError.INVALID_VALUE` error. CONCAT = 0b10; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// A very basic key-value store. @discoverable open protocol Store { /// Writes an item to the store. /// /// Since the value stored in the key-value store can now be different from the input (if the /// `WriteOptions.CONCAT` flag is set), we need to return the resulting `Value` to the /// requester. /// /// We use an (anonymous) `table` and a (named) `flexible union` as the request and response /// payload, respectively, to allow for easier future evolution. Both of these types are /// `flexible`, meaning that adding or removing members is binary-compatible. This makes them /// much easier to evolve that the `struct` types that were previously used, which cannot be /// changed after release without breaking ABI. flexible WriteItem(table { 1: attempt Item; 2: options WriteOptions; }) -> (Value) error WriteError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store" }, ], config: { // A vector of values for every easily representible type in our key-value store. For // brevity's sake, the 8, 16, and 32 bit integer types and booleans are omitted. // // TODO(https://fxbug.dev/42178362): It would absolve individual language implementations of a great // deal of string parsing if we were able to use all FIDL constructs directly here. In // particular, floats and nested types are very difficult to represent, and have been // excluded from this example for the time being. set_concat_option: { type: "bool" }, set_overwrite_option: { type: "bool" }, write_bytes: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, write_strings: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, write_uint64s: { type: "vector", max_count: 16, element: { type: "uint64" }, }, write_int64s: { type: "vector", max_count: 16, element: { type: "int64" }, }, // Note: due to the limitation of structured config not allowing vectors nested in vectors, // we only set the lower half of the uint128 for simplicity's sake. write_uint128s: { type: "vector", max_count: 16, element: { type: "uint64" }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.usegenericvalues.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, config::Config, fidl_examples_keyvaluestore_usegenericvalues::{ Item, StoreMarker, StoreProxy, StoreWriteItemRequest, Value, WriteOptions, }, fuchsia_component::client::connect_to_protocol, std::{thread, time}, }; // A helper function to sequentially write a single item to the key-value store and print a log when // successful. async fn write_next_item( store: &StoreProxy, key: &str, value: Value, options: WriteOptions, ) -> Result<(), Error> { // Create an empty request payload using `::default()`. let mut req = StoreWriteItemRequest::default(); req.options = Some(options); // Fill in the `Item` we will be attempting to write. println!("WriteItem request sent: key: {}, value: {:?}", &key, &value); req.attempt = Some(Item { key: key.to_string(), value: value }); // Send and async `WriteItem` request to the server. match store.write_item(&req).await.context("Error sending request")? { Ok(value) => println!("WriteItem response received: {:?}", &value), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } Ok(()) } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // All of our requests will have the same bitflags set. Pull these settings from the config. let mut options = WriteOptions::empty(); options.set(WriteOptions::OVERWRITE, config.set_overwrite_option); options.set(WriteOptions::CONCAT, config.set_concat_option); // The structured config provides one input for most data types that can be stored in the data // store. Iterate through those inputs in the order we see them in the FIDL file. // // Note that FIDL unions are rendered as enums in Rust; for example, the `Value` union has now // become a `Value` Rust enum, with each member taking exactly one argument. for value in config.write_bytes.into_iter() { write_next_item(&store, "bytes", Value::Bytes(value.into()), options).await?; } for value in config.write_strings.into_iter() { write_next_item(&store, "string", Value::String(value), options).await?; } for value in config.write_uint64s.into_iter() { write_next_item(&store, "uint64", Value::Uint64(value), options).await?; } for value in config.write_int64s.into_iter() { write_next_item(&store, "int64", Value::Int64(value), options).await?; } for value in config.write_uint128s.into_iter() { write_next_item(&store, "uint128", Value::Uint128([0, value]), options).await?; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fuchsia_component::server::ServiceFs; use futures::prelude::*; use lazy_static::lazy_static; use regex::Regex; use std::cell::RefCell; use std::collections::hash_map::Entry; use std::collections::HashMap; use fidl_examples_keyvaluestore_usegenericvalues::{ Item, StoreRequest, StoreRequestStream, Value, WriteError, WriteOptions, }; use std::collections::hash_map::OccupiedEntry; use std::ops::Add; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } /// Sums any numeric type. fn sum<T: Add + Add<Output = T> + Copy>(operands: [T; 2]) -> T { operands[0] + operands[1] } /// Clones and inserts an entry, so that the original (now concatenated) copy may be returned in the /// response. fn write(inserting: Value, mut entry: OccupiedEntry<'_, String, Value>) -> Value { entry.insert(inserting.clone()); println!("Wrote key: {}, value: {:?}", entry.key(), &inserting); inserting } /// Handler for the `WriteItem` method. fn write_item( store: &mut HashMap<String, Value>, attempt: Item, options: &WriteOptions, ) -> Result<Value, WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY for key: {}", attempt.key); return Err(WriteError::InvalidKey); } match store.entry(attempt.key) { Entry::Occupied(entry) => { // The `CONCAT` flag supersedes the `OVERWRITE` flag, so check it first. if options.contains(WriteOptions::CONCAT) { match entry.get() { Value::Bytes(old) => { if let Value::Bytes(new) = attempt.value { let mut combined = old.clone(); combined.extend(new); return Ok(write(Value::Bytes(combined), entry)); } } Value::String(old) => { if let Value::String(new) = attempt.value { return Ok(write(Value::String(format!("{}{}", old, &new)), entry)); } } Value::Uint64(old) => { if let Value::Uint64(new) = attempt.value { return Ok(write(Value::Uint64(sum([*old, new])), entry)); } } Value::Int64(old) => { if let Value::Int64(new) = attempt.value { return Ok(write(Value::Int64(sum([*old, new])), entry)); } } // Note: only works on the uint64 range in practice. Value::Uint128(old) => { if let Value::Uint128(new) = attempt.value { return Ok(write(Value::Uint128([0, sum([old[1], new[1]])]), entry)); } } _ => { panic!("actively unsupported type!") } } // Only reachable if the type of the would be concatenated value did not match the // value already occupying this entry. println!("Write error: INVALID_VALUE for key: {}", entry.key()); return Err(WriteError::InvalidValue); } // If we're not doing CONCAT, check for OVERWRITE next. if options.contains(WriteOptions::OVERWRITE) { return Ok(write(attempt.value, entry)); } println!("Write error: ALREADY_EXISTS for key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote key: {}, value: {:?}", entry.key(), &attempt.value); entry.insert(attempt.value.clone()); Ok(attempt.value) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, Value>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { // Because we are using a table payload, there is an extra level of indirection. The // top-level container for the table itself is always called "payload". StoreRequest::WriteItem { payload, responder } => { println!("WriteItem request received"); // Error out if either of the request table's members are not set. let attempt = payload.attempt.context("required field 'attempt' is unset")?; let options = payload.options.context("required field 'options' is unset")?; // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send( write_item(&mut store.borrow_mut(), attempt, &options) .as_ref() .map_err(|e| *e), ) .context("error sending reply")?; println!("WriteItem response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Discoverable
FIDL recipe: @discoverable
The @discoverable attribute assigns a name for service discovery. This allows the client to search for the correct name without the need to manually ensure the lookup name matches the one passed on the server side.
In this example, you will create a basic calculator server & client which shows the fundamental setup needed to first define and then serve and consume a FIDL protocol.
First, you will define the interface definitions and test harness. The
interface definition (the .fidl file itself) is the starting point for any new
FIDL protocol. Additionally, the calculator includes the necessary CML and realm
definitions to create a client-server pattern which can be used as project
scaffolding for arbitrary implementations.
See below for the FIDL code:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. // The namespace for this FIDL protocol. This namespace is how both consumers (clients) and providers (servers) reference this protocol. library examples.calculator.baseline; // @discoverable indicates 'Calculator' is a protocol that will be served under the examples.calculator.baseline libarary namespace. https://fuchsia.dev/fuchsia-src/reference/fidl/language/attributes#discoverable . If @discoverable is missing, it will lead to a compile time error when trying to import the library. @discoverable // A limited-functionality calculator 'protocol' that adds and subtracts integers. open protocol Calculator { // Takes as input a struct with two integers, and returns their sum: (a+b)=sum. This method is infallible (no errors can be generated) as two int32's cannot overflow a result type of int64. flexible Add(struct { a int32; b int32; }) -> (struct { sum int64; }); // Takes as input a struct with two integers, and returns their difference: (a-b)=difference. This method is infallible (no errors can be generated) as two int32's cannot overflow a result type of int64. flexible Subtract(struct { a int32; b int32; }) -> (struct { difference int64; }); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.calculator.baseline.Calculator" }, ], config: {}, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.calculator.baseline.Calculator" }, ], expose: [ { protocol: "examples.calculator.baseline.Calculator", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.calculator.baseline.Calculator", from: "#server", to: "#client", }, // Route logging support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// TODO(https://fxbug.dev/42063075): Rust implementation.Server
// TODO(https://fxbug.dev/42063075): Rust implementation.C++ (Natural)
Client
// TODO(https://fxbug.dev/42063075): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42063075): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42063075): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42063075): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42063075): HLCPP implementation.Server
// TODO(https://fxbug.dev/42063075): HLCPP implementation.Creating a FIDL protocol from the ground up as is shown in this example can be a more common scenario for certain developers, such as platform developers. However, other types of developers also benefit from learning how to construct a FIDL protocol even if they won't typically do so. This helps you learn how everything about FIDL fits together, including the syntax, grammar, language features, how to serve and consume a given FIDL protocol, and how the build system works. For next steps, the examples which follow this baseline show how to extend an existing FIDL protocol, which is expected to be a fairly common practice.
Enum
FIDL recipe: Enum
An enum is a FIDL data type that represents a fixed list of possible constants, like the suits in a deck of playing cards, or the make of car a user may select from a dropdown menu. This list of values is then mapped over an underlying integer type, with each value thereof corresponding to one of the listed members.
In the example below, a FIDL enum is added in a scenario where enums are a
perfect fit: enumerating the possible error values that may be emitted by a
failed method call. The ReadError enum has two members: NOT_FOUND is used to
indicate that a search key could not be matched during a read attempt, while
UNKNOWN serves as a grab-bag error for all cases that cannot be explicitly
described. Note that this enum is marked flexible, allowing it to be easily
evolved with new members in the future.
Reasoning
The original write-only key-value store is now extended with the ability to read items back out of the store.
Implementation
The changes applied to the FIDL and CML definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.addreaditem; // Aliases for the key and value. Using aliases helps increase the readability of FIDL files and // reduces likelihood of errors due to differing constraints. alias Key = string:128; alias Value = vector<byte>:64000; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key Key; value Value; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// An enumeration of things that may go wrong when trying to read a value out of our store. type ReadError = flexible enum { UNKNOWN = 0; NOT_FOUND = 1; }; /// A very basic key-value store - so basic, in fact, that one may only write to it, never read! @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; /// Reads an item from the store. flexible ReadItem(struct { key Key; }) -> (Item) error ReadError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.addreaditem.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, read_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.addreaditem.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.addreaditem.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.addreaditem.Store", from: "#server", to: "#client", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations for all languages change as well:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use config::Config; use fidl_examples_keyvaluestore_addreaditem::{Item, StoreMarker}; use fuchsia_component::client::connect_to_protocol; use std::{str, thread, time}; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; match store.write_item(&Item { key: key, value: value.into_bytes() }).await? { Ok(_) => println!("WriteItem Success"), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // The structured config for this client contains `read_items`, a vector of strings, each of // which is meant to be read from the key-value store. We iterate over these keys, attempting to // read them in turn. for key in config.read_items.into_iter() { let res = store.read_item(key.as_str()).await; match res.unwrap() { Ok(val) => { println!("ReadItem Success: key: {}, value: {}", key, str::from_utf8(&val.1)?) } Err(err) => println!("ReadItem Error: {}", err.into_primitive()), } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, fidl_examples_keyvaluestore_addreaditem::{ Item, ReadError, StoreRequest, StoreRequestStream, WriteError, }, fuchsia_component::server::ServiceFs, futures::prelude::*, lazy_static::lazy_static, regex::Regex, std::cell::RefCell, std::collections::hash_map::Entry, std::collections::HashMap, }; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z][A-Za-z0-9_\./]{2,62}[A-Za-z0-9]$") .expect("Key validation regex failed to compile"); } /// Handler for the `WriteItem` method. fn write_item(store: &mut HashMap<String, Vec<u8>>, attempt: Item) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Validate the value. if attempt.value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", attempt.key); return Err(WriteError::InvalidValue); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote value at key: {}", entry.key()); entry.insert(attempt.value); Ok(()) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, Vec<u8>>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.borrow_mut(), attempt)) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::ReadItem { key, responder } => { println!("ReadItem request received"); // Read the item from the store, returning the appropriate error if it could not be found. responder .send(match store.borrow().get(&key) { Some(found) => { println!("Read value at key: {}", key); Ok((&key, found)) } None => { println!("Read error: NOT_FOUND, For key: {}", key); Err(ReadError::NotFound) } }) .context("error sending reply")?; println!("ReadItem response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Feed forward pattern
FIDL recipe: Feed forward pattern
In FIDL protocols, using two-way methods with empty responses for flow control
purposes (e.g. DoSomething(...) -> ();) has a fundamental drawback for latency
sensitive protocols: either callers wait for each reply before sending the next
message, thereby increasing per-message latency, or they ignore it, which
renders the empty reply itself pointless. For protocols that desire flow control
without these latency costs, a good alternative is the feed forward pattern.
In this setup, one or more one-way methods write data to the server, while some
other method (one-way or two-way) is used to "commit" the work and synchronize
between the client and server. This means arbitrary amounts of data can be
transferred as fast as they can be sent, but there is still some amount of flow
control, as the synchronization method forces the client to stop before
proceeding with more work.
One way to improve the performance of the Instance protocol is to allow the
batching of lines: rather than sending a single AddLine(...); every time we
have a new line we'd like to add to the canvas, waiting for the reply, then
doing it again for the next line, we can instead batch many lines into a single
invocation of the new AddLines(...); call. The client can now decide how to
best segment a large collection of lines to be drawn.
Naively implemented, we would find ourselves in a situation where the server and
the client are completely unsynchronized: the client can flood the server with
unbounded AddLines(...); calls, and the server can similarly flood the client
with more -> OnDrawn(...); events than it can handle. The solution to both of
these problems is to add a simple Ready() -> (); method for synchronization
purposes. This method is called by the client whenever it is prepared to receive
the next draw update, with the response from the server indicating that the client
can proceed with more requests.
We now have some flow control in both directions. The protocol now implements
the feed forward pattern, allowing many uncontrolled calls before some
synchronizing "commit" call triggers the actual work on the server. This
prevents the client from overwhelming the server with work. Similarly, the
server is no longer allowed to send unbounded -> OnDrawn(...); events: each
event must follow a signal from the client, the Ready() -> (); call, that
indicates that it is ready to do more work. This is known as the throttled
event pattern.
The concrete implementations must apply some of these rules manually: the client
must close the connection if it receives an -> OnDrawn(...); event it did not
request via the Ready() -> (); method.
The FIDL, CML, and realm interface definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.canvas.clientrequesteddraw; /// A point in 2D space. type Point = struct { x int64; y int64; }; /// A line in 2D space. alias Line = array<Point, 2>; /// A bounding box in 2D space. This is the result of "drawing" operations on our canvas, and what /// the server reports back to the client. These bounds are sufficient to contain all of the /// lines (inclusive) on a canvas at a given time. type BoundingBox = struct { top_left Point; bottom_right Point; }; /// Manages a single instance of a canvas. Each session of this protocol is responsible for a new /// canvas. @discoverable open protocol Instance { /// Add multiple lines to the canvas. We are able to reduce protocol chatter and the number of /// requests needed by batching instead of calling the simpler `AddLine(...)` one line at a /// time. flexible AddLines(struct { lines vector<Line>; }); /// Rather than the server randomly performing draws, or trying to guess when to do so, the /// client must explicitly ask for them. This creates a bit of extra chatter with the additional /// method invocation, but allows much greater client-side control of when the canvas is "ready" /// for a view update, thereby eliminating unnecessary draws. /// /// This method also has the benefit of "throttling" the `-> OnDrawn(...)` event - rather than /// allowing a potentially unlimited flood of `-> OnDrawn(...)` calls, we now have the runtime /// enforced semantic that each `-> OnDrawn(...)` call must follow a unique `Ready() -> ()` call /// from the client. An unprompted `-> OnDrawn(...)` is invalid, and should cause the channel to /// immediately close. flexible Ready() -> (); /// Update the client with the latest drawing state. The server makes no guarantees about how /// often this event occurs - it could occur multiple times per board state, for example. flexible -> OnDrawn(BoundingBox); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.canvas.clientrequesteddraw.Instance" }, ], config: { // A script for the client to follow. Entries in the script may take one of two forms: a // pair of signed-integer coordinates like "-2,15:4,5", or the string "READY". The former // builds a local vector sent via a single `AddLines(...)` call, while the latter sends a // `Ready() -> ()` call pauses execution until the next `->OnDrawn(...)` event is received. // // TODO(https://fxbug.dev/42178362): It would absolve individual language implementations of a great // deal of string parsing if we were able to use a vector of `union { Point; Ready}` here. script: { type: "vector", max_count: 100, element: { type: "string", max_size: 64, }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.canvas.clientrequesteddraw.Instance" }, ], expose: [ { protocol: "examples.canvas.clientrequesteddraw.Instance", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.canvas.clientrequesteddraw.Instance", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{format_err, Context as _, Error}; use config::Config; use fidl_examples_canvas_clientrequesteddraw::{InstanceEvent, InstanceMarker, Point}; use fuchsia_component::client::connect_to_protocol; use futures::TryStreamExt; use std::{thread, time}; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send Instance requests // across the channel. let instance = connect_to_protocol::<InstanceMarker>()?; println!("Outgoing connection enabled"); let mut batched_lines = Vec::<[Point; 2]>::new(); for action in config.script.into_iter() { // If the next action in the script is to "PUSH", send a batch of lines to the server. if action == "PUSH" { instance.add_lines(&batched_lines).context("Could not send lines")?; println!("AddLines request sent"); batched_lines.clear(); continue; } // If the next action in the script is to "WAIT", block until an OnDrawn event is received // from the server. if action == "WAIT" { let mut event_stream = instance.take_event_stream(); loop { match event_stream .try_next() .await .context("Error getting event response from proxy")? .ok_or_else(|| format_err!("Proxy sent no events"))? { InstanceEvent::OnDrawn { top_left, bottom_right } => { println!( "OnDrawn event received: top_left: {:?}, bottom_right: {:?}", top_left, bottom_right ); break; } InstanceEvent::_UnknownEvent { ordinal, .. } => { println!("Received an unknown event with ordinal {ordinal}"); } } } // Now, inform the server that we are ready to receive more updates whenever they are // ready for us. println!("Ready request sent"); instance.ready().await.context("Could not send ready call")?; println!("Ready success"); continue; } // Add a line to the next batch. Parse the string input, making two points out of it. let mut points = action .split(":") .map(|point| { let integers = point .split(",") .map(|integer| integer.parse::<i64>().unwrap()) .collect::<Vec<i64>>(); Point { x: integers[0], y: integers[1] } }) .collect::<Vec<Point>>(); // Assemble a line from the two points. let from = points.pop().ok_or_else(|| format_err!("line requires 2 points, but has 0"))?; let to = points.pop().ok_or_else(|| format_err!("line requires 2 points, but has 1"))?; let mut line: [Point; 2] = [from, to]; // Batch a line for drawing to the canvas using the two points provided. println!("AddLines batching line: {:?}", &mut line); batched_lines.push(line); } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{anyhow, Context as _, Error}; use fidl::endpoints::RequestStream as _; use fidl_examples_canvas_clientrequesteddraw::{ BoundingBox, InstanceRequest, InstanceRequestStream, Point, }; use fuchsia_async::{MonotonicInstant, Timer}; use fuchsia_component::server::ServiceFs; use futures::future::join; use futures::prelude::*; use std::sync::{Arc, Mutex}; // A struct that stores the two things we care about for this example: the bounding box the lines // that have been added thus far, and bit to track whether or not there have been changes since the // last `OnDrawn` event. #[derive(Debug)] struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. changed: bool, // Tracks whether or not the client has declared itself ready to receive more updated. ready: bool, bounding_box: BoundingBox, } /// Handler for the `AddLines` method. fn add_lines(state: &mut CanvasState, lines: Vec<[Point; 2]>) { // Update the bounding box to account for the new lines we've just "added" to the canvas. let bounds = &mut state.bounding_box; for line in lines { println!("AddLines printing line: {:?}", line); for point in line { if point.x < bounds.top_left.x { bounds.top_left.x = point.x; } if point.y > bounds.top_left.y { bounds.top_left.y = point.y; } if point.x > bounds.bottom_right.x { bounds.bottom_right.x = point.x; } if point.y < bounds.bottom_right.y { bounds.bottom_right.y = point.y; } } } // Mark the state as "dirty", so that an update is sent back to the client on the next tick. state.changed = true } /// Creates a new instance of the server, paired to a single client across a zircon channel. async fn run_server(stream: InstanceRequestStream) -> Result<(), Error> { // Create a new in-memory state store for the state of the canvas. The store will live for the // lifetime of the connection between the server and this particular client. let state = Arc::new(Mutex::new(CanvasState { changed: true, ready: true, bounding_box: BoundingBox { top_left: Point { x: 0, y: 0 }, bottom_right: Point { x: 0, y: 0 }, }, })); // Take ownership of the control_handle from the stream, which will allow us to push events from // a different async task. let control_handle = stream.control_handle(); // A separate watcher task periodically "draws" the canvas, and notifies the client of the new // state. We'll need a cloned reference to the canvas state to be accessible from the new // task. let state_ref = state.clone(); let update_sender = || async move { loop { // Our server sends one update per second, but only if the client has declared that it // is ready to receive one. Timer::new(MonotonicInstant::after(zx::MonotonicDuration::from_seconds(1))).await; let mut state = state_ref.lock().unwrap(); if !state.changed || !state.ready { continue; } // After acquiring the lock, this is where we would draw the actual lines. Since this is // just an example, we'll avoid doing the actual rendering, and simply send the bounding // box to the client instead. let bounds = state.bounding_box; match control_handle.send_on_drawn(&bounds.top_left, &bounds.bottom_right) { Ok(_) => println!( "OnDrawn event sent: top_left: {:?}, bottom_right: {:?}", bounds.top_left, bounds.bottom_right ), Err(_) => return, } // Reset the change and ready trackers. state.ready = false; state.changed = false; } }; // Handle requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. let state_ref = &state; let request_handler = stream.map(|result| result.context("failed request")).try_for_each(|request| async move { // Match based on the method being invoked. match request { InstanceRequest::AddLines { lines, .. } => { println!("AddLines request received"); add_lines(&mut state_ref.lock().unwrap(), lines); } InstanceRequest::Ready { responder, .. } => { println!("Ready request received"); // The client must only call `Ready() -> ();` after receiving an `-> OnDrawn();` // event; if two "consecutive" `Ready() -> ();` calls are received, this // interaction has entered an invalid state, and should be aborted immediately. let mut state = state_ref.lock().unwrap(); if state.ready == true { return Err(anyhow!("Invalid back-to-back `Ready` requests received")); } state.ready = true; responder.send().context("Error responding")?; } // InstanceRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }); // This line will only be reached if the server errors out. The stream will await indefinitely, // thereby creating a long-lived server. Here, we first wait for the updater task to realize the // connection has died, then bubble up the error. join(request_handler, update_sender()).await.0 } // A helper enum that allows us to treat a `Instance` service instance as a value. enum IncomingService { Instance(InstanceRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Instance` protocol - this will allow the client to see // the server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Instance); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Instance(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.clientrequesteddraw/cpp/fidl.h> #include <lib/async-loop/cpp/loop.h> #include <lib/component/incoming/cpp/protocol.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <charconv> #include <examples/fidl/new/canvas/client_requested_draw/cpp_natural/client/config.h> // The |EventHandler| is a derived class that we pass into the |fidl::WireClient| to handle incoming // events asynchronously. class EventHandler : public fidl::AsyncEventHandler<examples_canvas_clientrequesteddraw::Instance> { public: // Handler for |OnDrawn| events sent from the server. void OnDrawn( fidl::Event<examples_canvas_clientrequesteddraw::Instance::OnDrawn>& event) override { ::examples_canvas_clientrequesteddraw::Point top_left = event.top_left(); ::examples_canvas_clientrequesteddraw::Point bottom_right = event.bottom_right(); FX_LOGS(INFO) << "OnDrawn event received: top_left: Point { x: " << top_left.x() << ", y: " << top_left.y() << " }, bottom_right: Point { x: " << bottom_right.x() << ", y: " << bottom_right.y() << " }"; loop_.Quit(); } void on_fidl_error(fidl::UnbindInfo error) override { FX_LOGS(ERROR) << error; } void handle_unknown_event( fidl::UnknownEventMetadata<examples_canvas_clientrequesteddraw::Instance> metadata) override { FX_LOGS(WARNING) << "Received an unknown event with ordinal " << metadata.event_ordinal; } explicit EventHandler(async::Loop& loop) : loop_(loop) {} private: async::Loop& loop_; }; // A helper function that takes a coordinate in string form, like "123,-456", and parses it into a // a struct of the form |{ in64 x; int64 y; }|. ::examples_canvas_clientrequesteddraw::Point ParsePoint(std::string_view input) { int64_t x = 0; int64_t y = 0; size_t index = input.find(','); if (index != std::string::npos) { std::from_chars(input.data(), input.data() + index, x); std::from_chars(input.data() + index + 1, input.data() + input.length(), y); } return ::examples_canvas_clientrequesteddraw::Point(x, y); } using Line = ::std::array<::examples_canvas_clientrequesteddraw::Point, 2>; // A helper function that takes a coordinate pair in string form, like "1,2:-3,-4", and parses it // into an array of 2 |Point| structs. Line ParseLine(const std::string& action) { auto input = std::string_view(action); size_t index = input.find(':'); if (index != std::string::npos) { return {ParsePoint(input.substr(0, index)), ParsePoint(input.substr(index + 1))}; } return {}; } int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop and dispatcher. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace. This can fail so it's wrapped in a // |zx::result| and it must be checked for errors. zx::result client_end = component::Connect<examples_canvas_clientrequesteddraw::Instance>(); if (!client_end.is_ok()) { FX_LOGS(ERROR) << "Synchronous error when connecting to the |Instance| protocol: " << client_end.status_string(); return -1; } // Create an instance of the event handler. EventHandler event_handler(loop); // Create an asynchronous client using the newly-established connection. fidl::Client client(std::move(*client_end), dispatcher, &event_handler); FX_LOGS(INFO) << "Outgoing connection enabled"; std::vector<Line> batched_lines; for (const auto& action : conf.script()) { // If the next action in the script is to "PUSH", send a batch of lines to the server. if (action == "PUSH") { fit::result<fidl::Error> result = client->AddLines(batched_lines); if (!result.is_ok()) { // Check that our one-way call was enqueued successfully, and handle the error // appropriately. In the case of this example, there is nothing we can do to recover here, // except to log an error and exit the program. FX_LOGS(ERROR) << "Could not send AddLines request: " << result.error_value(); return -1; } batched_lines.clear(); FX_LOGS(INFO) << "AddLines request sent"; continue; } // If the next action in the script is to "WAIT", block until an |OnDrawn| event is received // from the server. if (action == "WAIT") { loop.Run(); loop.ResetQuit(); // Now, inform the server that we are ready to receive more updates whenever they are // ready for us. FX_LOGS(INFO) << "Ready request sent"; client->Ready().ThenExactlyOnce( [&](fidl::Result<examples_canvas_clientrequesteddraw::Instance::Ready> result) { // Check if the FIDL call succeeded or not. if (result.is_ok()) { FX_LOGS(INFO) << "Ready success"; } else { FX_LOGS(ERROR) << "Could not send Ready request: " << result.error_value(); } // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); continue; } // Batch a line for drawing to the canvas using the two points provided. Line line = ParseLine(action); batched_lines.push_back(line); FX_LOGS(INFO) << "AddLines batching line: [Point { x: " << line[1].x() << ", y: " << line[1].y() << " }, Point { x: " << line[0].x() << ", y: " << line[0].y() << " }]"; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.clientrequesteddraw/cpp/fidl.h> #include <lib/async-loop/cpp/loop.h> #include <lib/async/cpp/task.h> #include <lib/component/outgoing/cpp/outgoing_directory.h> #include <lib/fidl/cpp/wire/channel.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <src/lib/fxl/macros.h> #include <src/lib/fxl/memory/weak_ptr.h> // A struct that stores the two things we care about for this example: the set of lines, and the // bounding box that contains them. struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. bool changed = true; // Tracks whether or not the client has declared itself ready to receive more updated. bool ready = true; examples_canvas_clientrequesteddraw::BoundingBox bounding_box; }; // An implementation of the |Instance| protocol. class InstanceImpl final : public fidl::Server<examples_canvas_clientrequesteddraw::Instance> { public: // Bind this implementation to a channel. InstanceImpl(async_dispatcher_t* dispatcher, fidl::ServerEnd<examples_canvas_clientrequesteddraw::Instance> server_end) : binding_(dispatcher, std::move(server_end), this, std::mem_fn(&InstanceImpl::OnFidlClosed)), weak_factory_(this) { // Start the update timer on startup. Our server sends one update per second ScheduleOnDrawnEvent(dispatcher, zx::sec(1)); } void OnFidlClosed(fidl::UnbindInfo info) { if (info.reason() != ::fidl::Reason::kPeerClosedWhileReading) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; } void AddLines(AddLinesRequest& request, AddLinesCompleter::Sync& completer) override { FX_LOGS(INFO) << "AddLines request received"; for (const auto& points : request.lines()) { FX_LOGS(INFO) << "AddLines printing line: [Point { x: " << points[1].x() << ", y: " << points[1].y() << " }, Point { x: " << points[0].x() << ", y: " << points[0].y() << " }]"; // Update the bounding box to account for the new line we've just "added" to the canvas. auto& bounds = state_.bounding_box; for (const auto& point : points) { if (point.x() < bounds.top_left().x()) { bounds.top_left().x() = point.x(); } if (point.y() > bounds.top_left().y()) { bounds.top_left().y() = point.y(); } if (point.x() > bounds.bottom_right().x()) { bounds.bottom_right().x() = point.x(); } if (point.y() < bounds.bottom_right().y()) { bounds.bottom_right().y() = point.y(); } } } // Mark the state as "dirty", so that an update is sent back to the client on the next |OnDrawn| // event. state_.changed = true; } void Ready(ReadyCompleter::Sync& completer) override { FX_LOGS(INFO) << "Ready request received"; // The client must only call `Ready() -> ();` after receiving an `-> OnDrawn();` event; if two // "consecutive" `Ready() -> ();` calls are received, this interaction has entered an invalid // state, and should be aborted immediately. if (state_.ready == true) { FX_LOGS(ERROR) << "Invalid back-to-back `Ready` requests received"; } state_.ready = true; completer.Reply(); } void handle_unknown_method( fidl::UnknownMethodMetadata<examples_canvas_clientrequesteddraw::Instance> metadata, fidl::UnknownMethodCompleter::Sync& completer) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << metadata.method_ordinal; } private: // Each scheduled update waits for the allotted amount of time, sends an update if something has // changed, and schedules the next update. void ScheduleOnDrawnEvent(async_dispatcher_t* dispatcher, zx::duration after) { async::PostDelayedTask( dispatcher, [&, dispatcher, after, weak = weak_factory_.GetWeakPtr()] { // Halt execution if the binding has been deallocated already. if (!weak) { return; } // Schedule the next update if the binding still exists. weak->ScheduleOnDrawnEvent(dispatcher, after); // No need to send an update if nothing has changed since the last one, or the client has // not yet informed us that it is ready for more updates. if (!weak->state_.changed || !weak->state_.ready) { return; } // This is where we would draw the actual lines. Since this is just an example, we'll // avoid doing the actual rendering, and simply send the bounding box to the client // instead. auto result = fidl::SendEvent(binding_)->OnDrawn(state_.bounding_box); if (!result.is_ok()) { return; } auto top_left = state_.bounding_box.top_left(); auto bottom_right = state_.bounding_box.bottom_right(); FX_LOGS(INFO) << "OnDrawn event sent: top_left: Point { x: " << top_left.x() << ", y: " << top_left.y() << " }, bottom_right: Point { x: " << bottom_right.x() << ", y: " << bottom_right.y() << " }"; // Reset the change and ready trackers. state_.ready = false; state_.changed = false; }, after); } fidl::ServerBinding<examples_canvas_clientrequesteddraw::Instance> binding_; CanvasState state_ = CanvasState{}; // Generates weak references to this object, which are appropriate to pass into asynchronous // callbacks that need to access this object. The references are automatically invalidated // if this object is destroyed. fxl::WeakPtrFactory<InstanceImpl> weak_factory_; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. This // directory is where the outgoing FIDL protocols are installed so that they can be provided to // other components. component::OutgoingDirectory outgoing = component::OutgoingDirectory(dispatcher); // The `ServeFromStartupInfo()` function sets up the outgoing directory with the startup handle. // The startup handle is a handle provided to every component by the system, so that they can // serve capabilities (e.g. FIDL protocols) to other components. zx::result result = outgoing.ServeFromStartupInfo(); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to serve outgoing directory: " << result.status_string(); return -1; } // Register a handler for components trying to connect to // |examples.canvas.clientrequesteddraw.Instance|. result = outgoing.AddUnmanagedProtocol<examples_canvas_clientrequesteddraw::Instance>( [dispatcher](fidl::ServerEnd<examples_canvas_clientrequesteddraw::Instance> server_end) { // Create an instance of our InstanceImpl that destroys itself when the connection closes. new InstanceImpl(dispatcher, std::move(server_end)); }); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to add Instance protocol: " << result.status_string(); return -1; } // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
C++ (Wire)
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.clientrequesteddraw/cpp/wire.h> #include <lib/async-loop/cpp/loop.h> #include <lib/component/incoming/cpp/protocol.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <charconv> #include <examples/fidl/new/canvas/client_requested_draw/cpp_wire/client/config.h> // The |EventHandler| is a derived class that we pass into the |fidl::WireClient| to handle incoming // events asynchronously. class EventHandler : public fidl::WireAsyncEventHandler<examples_canvas_clientrequesteddraw::Instance> { public: // Handler for |OnDrawn| events sent from the server. void OnDrawn( fidl::WireEvent<examples_canvas_clientrequesteddraw::Instance::OnDrawn>* event) override { ::examples_canvas_clientrequesteddraw::wire::Point top_left = event->top_left; ::examples_canvas_clientrequesteddraw::wire::Point bottom_right = event->bottom_right; FX_LOGS(INFO) << "OnDrawn event received: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; loop_.Quit(); } void on_fidl_error(fidl::UnbindInfo error) override { FX_LOGS(ERROR) << error; } void handle_unknown_event( fidl::UnknownEventMetadata<examples_canvas_clientrequesteddraw::Instance> metadata) override { FX_LOGS(WARNING) << "Received an unknown event with ordinal " << metadata.event_ordinal; } explicit EventHandler(async::Loop& loop) : loop_(loop) {} private: async::Loop& loop_; }; // A helper function that takes a coordinate in string form, like "123,-456", and parses it into a // a struct of the form |{ in64 x; int64 y; }|. ::examples_canvas_clientrequesteddraw::wire::Point ParsePoint(std::string_view input) { int64_t x = 0; int64_t y = 0; size_t index = input.find(','); if (index != std::string::npos) { std::from_chars(input.data(), input.data() + index, x); std::from_chars(input.data() + index + 1, input.data() + input.length(), y); } return ::examples_canvas_clientrequesteddraw::wire::Point{.x = x, .y = y}; } using Line = ::fidl::Array<::examples_canvas_clientrequesteddraw::wire::Point, 2>; // A helper function that takes a coordinate pair in string form, like "1,2:-3,-4", and parses it // into an array of 2 |Point| structs. Line ParseLine(const std::string& action) { auto input = std::string_view(action); size_t index = input.find(':'); if (index != std::string::npos) { return {ParsePoint(input.substr(0, index)), ParsePoint(input.substr(index + 1))}; } return {}; } int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop and dispatcher. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace. This can fail so it's wrapped in a // |zx::result| and it must be checked for errors. zx::result client_end = component::Connect<examples_canvas_clientrequesteddraw::Instance>(); if (!client_end.is_ok()) { FX_LOGS(ERROR) << "Synchronous error when connecting to the |Instance| protocol: " << client_end.status_string(); return -1; } // Create an instance of the event handler. EventHandler event_handler(loop); // Create an asynchronous client using the newly-established connection. fidl::WireClient client(std::move(*client_end), dispatcher, &event_handler); FX_LOGS(INFO) << "Outgoing connection enabled"; std::vector<Line> batched_lines; for (const auto& action : conf.script()) { // If the next action in the script is to "PUSH", send a batch of lines to the server. if (action == "PUSH") { fidl::Status status = client->AddLines(fidl::VectorView<Line>::FromExternal(batched_lines)); if (!status.ok()) { // Check that our one-way call was enqueued successfully, and handle the error // appropriately. In the case of this example, there is nothing we can do to recover here, // except to log an error and exit the program. FX_LOGS(ERROR) << "Could not send AddLines request: " << status.error(); return -1; } batched_lines.clear(); FX_LOGS(INFO) << "AddLines request sent"; continue; } // If the next action in the script is to "WAIT", block until an |OnDrawn| event is received // from the server. if (action == "WAIT") { loop.Run(); loop.ResetQuit(); // Now, inform the server that we are ready to receive more updates whenever they are // ready for us. FX_LOGS(INFO) << "Ready request sent"; client->Ready().ThenExactlyOnce( [&](fidl::WireUnownedResult<examples_canvas_clientrequesteddraw::Instance::Ready>& result) { // Check if the FIDL call succeeded or not. if (result.ok()) { FX_LOGS(INFO) << "Ready success"; } else { FX_LOGS(ERROR) << "Could not send Ready request: " << result.error(); } // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); continue; } // Batch a line for drawing to the canvas using the two points provided. Line line = ParseLine(action); batched_lines.push_back(line); FX_LOGS(INFO) << "AddLines batching line: [Point { x: " << line[1].x << ", y: " << line[1].y << " }, Point { x: " << line[0].x << ", y: " << line[0].y << " }]"; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.clientrequesteddraw/cpp/wire.h> #include <lib/async-loop/cpp/loop.h> #include <lib/async/cpp/task.h> #include <lib/component/outgoing/cpp/outgoing_directory.h> #include <lib/fidl/cpp/wire/channel.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <src/lib/fxl/macros.h> #include <src/lib/fxl/memory/weak_ptr.h> // A struct that stores the two things we care about for this example: the set of lines, and the // bounding box that contains them. struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. bool changed = true; // Tracks whether or not the client has declared itself ready to receive more updated. bool ready = true; examples_canvas_clientrequesteddraw::wire::BoundingBox bounding_box; }; // An implementation of the |Instance| protocol. class InstanceImpl final : public fidl::WireServer<examples_canvas_clientrequesteddraw::Instance> { public: // Bind this implementation to a channel. InstanceImpl(async_dispatcher_t* dispatcher, fidl::ServerEnd<examples_canvas_clientrequesteddraw::Instance> server_end) : binding_(dispatcher, std::move(server_end), this, std::mem_fn(&InstanceImpl::OnFidlClosed)), weak_factory_(this) { // Start the update timer on startup. Our server sends one update per second ScheduleOnDrawnEvent(dispatcher, zx::sec(1)); } void OnFidlClosed(fidl::UnbindInfo info) { if (info.reason() != ::fidl::Reason::kPeerClosedWhileReading) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; } void AddLines(AddLinesRequestView request, AddLinesCompleter::Sync& completer) override { FX_LOGS(INFO) << "AddLines request received"; for (const auto& points : request->lines) { FX_LOGS(INFO) << "AddLines printing line: [Point { x: " << points[1].x << ", y: " << points[1].y << " }, Point { x: " << points[0].x << ", y: " << points[0].y << " }]"; // Update the bounding box to account for the new line we've just "added" to the canvas. auto& bounds = state_.bounding_box; for (const auto& point : points) { if (point.x < bounds.top_left.x) { bounds.top_left.x = point.x; } if (point.y > bounds.top_left.y) { bounds.top_left.y = point.y; } if (point.x > bounds.bottom_right.x) { bounds.bottom_right.x = point.x; } if (point.y < bounds.bottom_right.y) { bounds.bottom_right.y = point.y; } } } // Mark the state as "dirty", so that an update is sent back to the client on the next |OnDrawn| // event. state_.changed = true; } void Ready(ReadyCompleter::Sync& completer) override { FX_LOGS(INFO) << "Ready request received"; // The client must only call `Ready() -> ();` after receiving an `-> OnDrawn();` event; if two // "consecutive" `Ready() -> ();` calls are received, this interaction has entered an invalid // state, and should be aborted immediately. if (state_.ready == true) { FX_LOGS(ERROR) << "Invalid back-to-back `Ready` requests received"; } state_.ready = true; completer.Reply(); } void handle_unknown_method( fidl::UnknownMethodMetadata<examples_canvas_clientrequesteddraw::Instance> metadata, fidl::UnknownMethodCompleter::Sync& completer) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << metadata.method_ordinal; } private: // Each scheduled update waits for the allotted amount of time, sends an update if something has // changed, and schedules the next update. void ScheduleOnDrawnEvent(async_dispatcher_t* dispatcher, zx::duration after) { async::PostDelayedTask( dispatcher, [&, dispatcher, after, weak = weak_factory_.GetWeakPtr()] { // Halt execution if the binding has been deallocated already. if (!weak) { return; } // Schedule the next update if the binding still exists. weak->ScheduleOnDrawnEvent(dispatcher, after); // No need to send an update if nothing has changed since the last one, or the client has // not yet informed us that it is ready for more updates. if (!weak->state_.changed || !weak->state_.ready) { return; } // This is where we would draw the actual lines. Since this is just an example, we'll // avoid doing the actual rendering, and simply send the bounding box to the client // instead. auto top_left = weak->state_.bounding_box.top_left; auto bottom_right = weak->state_.bounding_box.bottom_right; fidl::Status status = fidl::WireSendEvent(weak->binding_)->OnDrawn(top_left, bottom_right); if (!status.ok()) { return; } FX_LOGS(INFO) << "OnDrawn event sent: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; // Reset the change and ready trackers. state_.ready = false; weak->state_.changed = false; }, after); } fidl::ServerBinding<examples_canvas_clientrequesteddraw::Instance> binding_; CanvasState state_ = CanvasState{}; // Generates weak references to this object, which are appropriate to pass into asynchronous // callbacks that need to access this object. The references are automatically invalidated // if this object is destroyed. fxl::WeakPtrFactory<InstanceImpl> weak_factory_; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. This // directory is where the outgoing FIDL protocols are installed so that they can be provided to // other components. component::OutgoingDirectory outgoing = component::OutgoingDirectory(dispatcher); // The `ServeFromStartupInfo()` function sets up the outgoing directory with the startup handle. // The startup handle is a handle provided to every component by the system, so that they can // serve capabilities (e.g. FIDL protocols) to other components. zx::result result = outgoing.ServeFromStartupInfo(); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to serve outgoing directory: " << result.status_string(); return -1; } // Register a handler for components trying to connect to // |examples.canvas.clientrequesteddraw.Instance|. result = outgoing.AddUnmanagedProtocol<examples_canvas_clientrequesteddraw::Instance>( [dispatcher](fidl::ServerEnd<examples_canvas_clientrequesteddraw::Instance> server_end) { // Create an instance of our InstanceImpl that destroys itself when the connection closes. new InstanceImpl(dispatcher, std::move(server_end)); }); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to add Instance protocol: " << result.status_string(); return -1; } // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
HLCPP
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <lib/async-loop/cpp/loop.h> #include <lib/sys/cpp/component_context.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <charconv> #include <examples/canvas/clientrequesteddraw/cpp/fidl.h> #include <examples/fidl/new/canvas/client_requested_draw/hlcpp/client/config.h> // A helper function that takes a coordinate in string form, like "123,-456", and parses it into a // a struct of the form |{ in64 x; int64 y; }|. ::examples::canvas::clientrequesteddraw::Point ParsePoint(std::string_view input) { int64_t x = 0; int64_t y = 0; size_t index = input.find(','); if (index != std::string::npos) { std::from_chars(input.data(), input.data() + index, x); std::from_chars(input.data() + index + 1, input.data() + input.length(), y); } return ::examples::canvas::clientrequesteddraw::Point{.x = x, .y = y}; } using Line = ::std::array<::examples::canvas::clientrequesteddraw::Point, 2>; // A helper function that takes a coordinate pair in string form, like "1,2:-3,-4", and parses it // into an array of 2 |Point| structs. Line ParseLine(const std::string& action) { auto input = std::string_view(action); size_t index = input.find(':'); if (index != std::string::npos) { return {ParsePoint(input.substr(0, index)), ParsePoint(input.substr(index + 1))}; } return {}; } int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace, then create an asynchronous client // using the newly-established connection. examples::canvas::clientrequesteddraw::InstancePtr instance_proxy; auto context = sys::ComponentContext::Create(); context->svc()->Connect(instance_proxy.NewRequest(dispatcher)); FX_LOGS(INFO) << "Outgoing connection enabled"; instance_proxy.set_error_handler([&loop](zx_status_t status) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; loop.Quit(); }); // Provide a lambda to handle incoming |OnDrawn| events asynchronously. instance_proxy.events().OnDrawn = [&loop](::examples::canvas::clientrequesteddraw::Point top_left, ::examples::canvas::clientrequesteddraw::Point bottom_right) { FX_LOGS(INFO) << "OnDrawn event received: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; loop.Quit(); }; instance_proxy.events().handle_unknown_event = [](uint64_t ordinal) { FX_LOGS(WARNING) << "Received an unknown event with ordinal " << ordinal; }; std::vector<Line> batched_lines; for (const auto& action : conf.script()) { // If the next action in the script is to "PUSH", send a batch of lines to the server. if (action == "PUSH") { instance_proxy->AddLines(batched_lines); batched_lines.clear(); FX_LOGS(INFO) << "AddLines request sent"; continue; } // If the next action in the script is to "WAIT", block until an |OnDrawn| event is received // from the server. if (action == "WAIT") { loop.Run(); loop.ResetQuit(); // Now, inform the server that we are ready to receive more updates whenever they are ready // for us. FX_LOGS(INFO) << "Ready request sent"; instance_proxy->Ready([&](fpromise::result<void, fidl::FrameworkErr> result) { if (result.is_error()) { // Check that our flexible two-way call was known to the server and handle the case of an // unknown method appropriately. In the case of this example, there is nothing we can do // to recover here, except to log an error and exit the program. FX_LOGS(ERROR) << "Server does not implement AddLine"; } FX_LOGS(INFO) << "Ready success"; // Quit the loop, thereby handing control back to the outer loop of actions being iterated // over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); continue; } // Batch a line for drawing to the canvas using the two points provided. Line line = ParseLine(action); batched_lines.push_back(line); FX_LOGS(INFO) << "AddLines batching line: [Point { x: " << line[1].x << ", y: " << line[1].y << " }, Point { x: " << line[0].x << ", y: " << line[0].y << " }]"; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <lib/async-loop/cpp/loop.h> #include <lib/async-loop/default.h> #include <lib/async/cpp/task.h> #include <lib/fidl/cpp/binding.h> #include <lib/sys/cpp/component_context.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <examples/canvas/clientrequesteddraw/cpp/fidl.h> #include <src/lib/fxl/macros.h> #include <src/lib/fxl/memory/weak_ptr.h> // A struct that stores the two things we care about for this example: the set of lines, and the // bounding box that contains them. struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. bool changed = true; // Tracks whether or not the client has declared itself ready to receive more updated. bool ready = true; examples::canvas::clientrequesteddraw::BoundingBox bounding_box; }; using Line = ::std::array<::examples::canvas::clientrequesteddraw::Point, 2>; // An implementation of the |Instance| protocol. class InstanceImpl final : public examples::canvas::clientrequesteddraw::Instance { public: // Bind this implementation to an |InterfaceRequest|. InstanceImpl(async_dispatcher_t* dispatcher, fidl::InterfaceRequest<examples::canvas::clientrequesteddraw::Instance> request) : binding_(fidl::Binding<examples::canvas::clientrequesteddraw::Instance>(this)), weak_factory_(this) { binding_.Bind(std::move(request), dispatcher); // Gracefully handle abrupt shutdowns. binding_.set_error_handler([this](zx_status_t status) mutable { if (status != ZX_ERR_PEER_CLOSED) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; }); // Start the update timer on startup. Our server sends one update per second. ScheduleOnDrawnEvent(dispatcher, zx::sec(1)); } void AddLines(std::vector<Line> lines) override { FX_LOGS(INFO) << "AddLines request received"; for (const auto& points : lines) { FX_LOGS(INFO) << "AddLines printing line: [Point { x: " << points[1].x << ", y: " << points[1].y << " }, Point { x: " << points[0].x << ", y: " << points[0].y << " }]"; // Update the bounding box to account for the new line we've just "added" to the canvas. auto& bounds = state_.bounding_box; for (const auto& point : points) { if (point.x < bounds.top_left.x) { bounds.top_left.x = point.x; } if (point.y > bounds.top_left.y) { bounds.top_left.y = point.y; } if (point.x > bounds.bottom_right.x) { bounds.bottom_right.x = point.x; } if (point.y < bounds.bottom_right.y) { bounds.bottom_right.y = point.y; } } } // Mark the state as "dirty", so that an update is sent back to the client on the next // |OnDrawn| event. state_.changed = true; } void Ready(ReadyCallback callback) override { FX_LOGS(INFO) << "Ready request received"; // The client must only call `Ready() -> ();` after receiving an `-> OnDrawn();` event; if // two "consecutive" `Ready() -> ();` calls are received, this interaction has entered an // invalid state, and should be aborted immediately. if (state_.ready == true) { FX_LOGS(ERROR) << "Invalid back-to-back `Ready` requests received"; } state_.ready = true; callback(fpromise::ok()); } void handle_unknown_method(uint64_t ordinal, bool method_has_response) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << ordinal; } private: // Each scheduled update waits for the allotted amount of time, sends an update if something // has changed, and schedules the next update. void ScheduleOnDrawnEvent(async_dispatcher_t* dispatcher, zx::duration after) { async::PostDelayedTask( dispatcher, [&, dispatcher, after, weak = weak_factory_.GetWeakPtr()] { // Halt execution if the binding has been deallocated already. if (!weak) { return; } // Schedule the next update if the binding still exists. weak->ScheduleOnDrawnEvent(dispatcher, after); // No need to send an update if nothing has changed since the last one, or the client // has not yet informed us that it is ready for more updates. if (!weak->state_.changed || !weak->state_.ready) { return; } // This is where we would draw the actual lines. Since this is just an example, we'll // avoid doing the actual rendering, and simply send the bounding box to the client // instead. auto top_left = state_.bounding_box.top_left; auto bottom_right = state_.bounding_box.bottom_right; binding_.events().OnDrawn(top_left, bottom_right); FX_LOGS(INFO) << "OnDrawn event sent: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; // Reset the change and ready trackers. state_.ready = false; state_.changed = false; }, after); } fidl::Binding<examples::canvas::clientrequesteddraw::Instance> binding_; CanvasState state_ = CanvasState{}; // Generates weak references to this object, which are appropriate to pass into asynchronous // callbacks that need to access this object. The references are automatically invalidated // if this object is destroyed. fxl::WeakPtrFactory<InstanceImpl> weak_factory_; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from // the client. The following initializes the loop, and obtains the dispatcher, which will be // used when binding the server implementation to a channel. // // Note that unlike the new C++ bindings, HLCPP bindings rely on the async loop being attached // to the current thread via the |kAsyncLoopConfigAttachToCurrentThread| configuration. async::Loop loop(&kAsyncLoopConfigAttachToCurrentThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. // This directory is where the outgoing FIDL protocols are installed so that they can be // provided to other components. auto context = sys::ComponentContext::CreateAndServeOutgoingDirectory(); // Register a handler for components trying to connect to // |examples.canvas.clientrequesteddraw.Instance|. context->outgoing()->AddPublicService( fidl::InterfaceRequestHandler<examples::canvas::clientrequesteddraw::Instance>( [dispatcher]( fidl::InterfaceRequest<examples::canvas::clientrequesteddraw::Instance> request) { // Create an instance of our |InstanceImpl| that destroys itself when the connection // closes. new InstanceImpl(dispatcher, std::move(request)); })); // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
Generated name
FIDL recipe: Generated name
A generated name is the FIDL compiler assigned name for an anonymous type. While anonymous types are not nameable in FIDL files themselves, there must be some name that refers to them in the generated bindings output, so end users may create instance of the type in the binding language.
Because the FIDL compiler name generation algorithm uses the local context
(member name, method name, and so on) to name the type, name collisions are
possible. To resolve such collisions, place an @generated_name attribute
directly before the type declaration, instructing the compiler which name it
should use instead.
In this variant, we allow our key-value store to take other key-value stores as
members. In short, we turn it into a tree. We do this by replacing the original
definition of value with one that utilizes a two-member union: one variant
stores leaf nodes using the same vector<byte> type as before, while the other
stores branch nodes in the form of other nested stores.
Reasoning
Here, we see several uses of optionality, whereby we can declare a type that may or may not exist. There are three flavors of optionality in FIDL:
- Types that have are always stored
out-of-line
on the wire, and thus have a builtin way to describe "absentness" via the
null envelope. Enabling
optionality for these types doesn't affect the wire shape of messages they are
included in - it simply changes which values are valid for that particular
type. The
union,vector<T>,client_end,server_end, andzx.Handletypes can all be made optional via the addition of the:optionalconstraint. By making ourvalueunionoptional, we are able to introduce a canonical "null" entry, in the form of an absentvalue. This means that emptybytesand absent/emptystoreproperties are invalid values. - Unlike the aforementioned types, the
structlayout has no extra space where a null header can be stored. Because of this, it needs to be wrapped in an envelope, changing the on-the-wire shape of the message it is being included in. To ensure that this wire-modifying effect easily legible, theItemstructtype must be wrapped in abox<T>type template. - Finally,
tablelayouts are always optional. An absenttableis simply one with none of its members set.
Trees are a naturally self-referential data structure: any node in the tree may
contain a leaf with pure data (in this case, a string), or a sub-tree with more
nodes. This requires recursion: the definition of Item is now transitively
dependent on itself! Representing recursive types in FIDL can be a bit tricky,
especially because support is currently somewhat
limited. We can support such types as long as there is
at least one optional type in the cycle created by the self-reference. For
instance, here we define the items struct member to be a box<Item>,
thereby breaking the includes cycle.
These changes also make heavy use of anonymous types, or types whose
declarations are inlined at their sole point of use, rather than being named,
top-level type declarations of their own. By default, the names of anonymous
types in the generated language bindings are taken from their local context. For
instance, the newly introduced flexible union takes on its owning member's
name Value, the newly introduced struct would become Store, and so on.
Because this heuristic can sometimes cause collisions, FIDL provides an escape
hatch by allowing the author to manually override an anonymous type's generated
name. This is done via the @generated_name attribute, which allows one to
change the name generated by backends. We can use one here, where the would-be
Store type is renamed to NestedStore to prevent a name collision with the
protocol declaration that uses that same name.
Implementation
The FIDL, CML, and realm interface definitions are modified as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.supporttrees; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value strict union { // Keep the original `bytes` as one of the options in the new union. All leaf nodes in the // tree must be `bytes`, or absent unions (representing empty). Empty byte arrays are // disallowed. 1: bytes vector<byte>:64000; // Allows a store within a store, thereby turning our flat key-value store into a tree // thereof. Note the use of `@generated_name` to prevent a type-name collision with the // `Store` protocol below, and the use of `box<T>` to ensure that there is a break in the // chain of recursion, thereby allowing `Item` to include itself in its own definition. // // This is a table so that added fields, like for example a `hash`, can be easily added in // the future. 2: store @generated_name("nested_store") table { 1: items vector<box<Item>>; }; }:optional; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// A very basic key-value store. @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.supporttrees.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A newline separated list nested entries. The first line should be the key // for the nested store, and each subsequent entry should be a pointer to a text file // containing the string value. The name of that text file (without the `.txt` suffix) will // serve as the entries key. write_nested: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A list of keys, all of which will be populated as null entries. write_null: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.supporttrees.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.supporttrees.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.supporttrees.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, config::Config, fidl_examples_keyvaluestore_supporttrees::{Item, NestedStore, StoreMarker, Value}, fuchsia_component::client::connect_to_protocol, std::{thread, time}, }; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; let res = store .write_item(&Item { key: key.clone(), value: Some(Box::new(Value::Bytes(value.into_bytes()))), }) .await; match res? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // Add nested entries to the key-value store as well. The entries are strings, where the first // line is the key of the entry, and each subsequent entry should be a pointer to a text file // containing the string value. The name of that text file (without the `.txt` suffix) will // serve as the entries key. for spec in config.write_nested.into_iter() { let mut items = vec![]; let mut nested_store = NestedStore::default(); let mut lines = spec.split("\n"); let key = lines.next().unwrap(); // For each entry, make a new entry in the `NestedStore` being built. for entry in lines { let path = format!("/pkg/data/{}.txt", entry); let contents = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; items.push(Some(Box::new(Item { key: entry.to_string(), value: Some(Box::new(Value::Bytes(contents.into()))), }))); } nested_store.items = Some(items); // Send the `NestedStore`, represented as a vector of values. let res = store .write_item(&Item { key: key.to_string(), value: Some(Box::new(Value::Store(nested_store))), }) .await; match res? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // Each entry in this list is a null value in the store. for key in config.write_null.into_iter() { match store.write_item(&Item { key: key.to_string(), value: None }).await? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. // Note: For the clarity of this example, allow code to be unused. #![allow(dead_code)] use { anyhow::{Context as _, Error}, fidl_examples_keyvaluestore_supporttrees::{ Item, StoreRequest, StoreRequestStream, Value, WriteError, }, fuchsia_component::server::ServiceFs, futures::prelude::*, lazy_static::lazy_static, regex::Regex, std::cell::RefCell, std::collections::hash_map::Entry, std::collections::HashMap, std::str::from_utf8, }; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } // A representation of a key-value store that can contain an arbitrarily deep nesting of other // key-value stores. #[allow(clippy::box_collection, reason = "mass allow for https://fxbug.dev/381896734")] enum StoreNode { Leaf(Option<Vec<u8>>), Branch(Box<HashMap<String, StoreNode>>), } /// Recursive item writer, which takes a `StoreNode` that may not necessarily be the root node, and /// writes an entry to it. fn write_item( store: &mut HashMap<String, StoreNode>, attempt: Item, path: &str, ) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { let key = format!("{}{}", &path, entry.key()); match attempt.value { // Null entries are allowed. None => { println!("Wrote value: NONE at key: {}", key); entry.insert(StoreNode::Leaf(None)); } Some(value) => match *value { // If this is a nested store, recursively make a new store to insert at this // position. Value::Store(entry_list) => { // Validate the value - absent stores, items lists with no children, or any // of the elements within that list being empty boxes, are all not allowed. if entry_list.items.is_some() { let items = entry_list.items.unwrap(); if !items.is_empty() && items.iter().all(|i| i.is_some()) { let nested_path = format!("{}/", key); let mut nested_store = HashMap::<String, StoreNode>::new(); for item in items.into_iter() { write_item(&mut nested_store, *item.unwrap(), &nested_path)?; } println!("Created branch at key: {}", key); entry.insert(StoreNode::Branch(Box::new(nested_store))); return Ok(()); } } println!("Write error: INVALID_VALUE, For key: {}", key); return Err(WriteError::InvalidValue); } // This is a simple leaf node on this branch. Value::Bytes(value) => { // Validate the value. if value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", key); return Err(WriteError::InvalidValue); } println!("Wrote key: {}, value: {:?}", key, from_utf8(&value).unwrap()); entry.insert(StoreNode::Leaf(Some(value))); } }, } Ok(()) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, StoreNode>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.borrow_mut(), attempt, "")) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Handle rights
FIDL recipe: Handle rights
A FIDL handle represents a unique capability in some system, usually the Zircon kernel. Handle rights are a FIDL-legible enumeration of the privileges that capability carries with it - for instance, whether or not the resource the handle represents may be written to, inspected, signalled, and so on.
Rights are validated at encode and decode time, ensuring that a given handle carries the set of privileges the interface author allotted for it.
A simple way to extend the key-value store to support exporting backups would be
to simply add a new method that stops the world, serializes the state of the
store, and sends it back as a FIDL vector<Item>. There are two downsides to
this approach, however. The first is that it puts all of the burden of the
backup on the server - a client pays nothing to ask for a backup operation that
is very expensive to the server. The second is that it involves a great deal of
copying: the client is almost certainly just going to write the resulting backup
to some backing datastore, like a file or a database, as soon as it receives it.
Having it decode this (potentially very large) FIDL object, just so that it can
immediately re-encode it as it forwards it to whatever protocol will do the
actual storage, is very wasteful.
Reasoning
A better solution is to use zircon's virtual memory objects. Instead of constantly copying bytes back and forth in a bucket brigade, we can mint a VMO to hold the backup data on the client, send it to the server, then forward it back to our target data store without deserializing in between. As long as the target data store's protocol has allowances for accepting data transported using a VMO, this is the preferred way to accomplish expensive operations like this. In fact, Fuchsia's file system, for instance, implements this exact pattern. A benefit of this approach is that it forces the client to do some work when asking the server for an expensive operation, minimizing the work imbalance between the two parties.
FIDL value types can be persisted to any byte-oriented storage medium, using
the FIDL data persistence binary format. We will persist the
newly introduced FIDL type Exportable into the VMO. The object will be encoded
and written to the storage (in this case, a VMO that could later be saved as a
file), and decoded from it when the data needs to be accessed again, in much the
same way that a message is encoded, transported, and decoded again later when
using FIDL over IPC.
To do this securely and adhere to the principle of least privilege,
we should constrain the privileges the handle representing our VMO may carry.
Enter handle rights, FIDL's first-class method of describing the privileges
available to a particular handle type. In this case, we allow the empty VMO
passed to the server in the Export request to be read from, queried for size,
resized, and written to. When the VMO is returned, we remove right to resize and
write, ensuring that no process, not even malicious actors in some far away
component, can modify this data as it moves through the system.
Implementation
The FIDL, CML, and realm interface definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.supportexports; using zx; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value vector<byte>:64000; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// An enumeration of things that may go wrong when trying to mint an export. type ExportError = flexible enum { UNKNOWN = 0; EMPTY = 1; STORAGE_TOO_SMALL = 2; }; // A data type describing the structure of a single export. We never actually send this data type // over the wire (we use the file's VMO instead), but whenever data needs to be written to/read from // its backing storage as persistent FIDL, it will have this schema. /// /// The items should be sorted in ascending order, following lexicographic ordering of their keys. type Exportable = table { 1: items vector<Item>; }; /// A very basic key-value store - so basic, in fact, that one may only write to it, never read! @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; /// Exports the entire store as a persistent [`Exportable`] FIDL object into a VMO provided by /// the client. /// /// By having the client provide (and speculatively size) the VMO, we force the party requesting /// the relatively heavy load of generating a backup to acknowledge and bear some of the costs. /// /// This method operates by having the client supply an empty VMO, which the server then /// attempts to fill. Notice that the server removes the `zx.Rights.WRITE` and /// `zx.Rights.SET_PROPERTY` rights from the returned VMO - not even the requesting client may /// alter the backup once it has been minted by the server. flexible Export(resource struct { /// Note that the empty VMO has more rights than the filled one being returned: it has /// `zx.Rights.WRITE` (via `zx.RIGHTS_IO`) so that the VMO may be filled with exported data, /// and `zx.Rights.SET_PROPERTY` (via `zx.RIGHTS_PROPERTY`) so that it may be resized to /// truncate any remaining empty buffer. empty zx.Handle:<VMO, zx.RIGHTS_BASIC | zx.RIGHTS_PROPERTY | zx.RIGHTS_IO>; }) -> (resource struct { /// The `zx.Rights.WRITE` and `zx.Rights.SET_PROPERTY` rights have been removed from the now /// filled VMO. No one, not even the client that requested the export, is able to modify /// this VMO going forward. filled zx.Handle:<VMO, zx.RIGHTS_BASIC | zx.Rights.GET_PROPERTY | zx.Rights.READ>; }) error ExportError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.supportexports.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // The size, in bytes, allotted to the export VMO max_export_size: { type: "uint64" }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.supportexports.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.supportexports.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.supportexports.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use config::Config; use fuchsia_component::client::connect_to_protocol; use std::{thread, time}; use fidl::unpersist; use fidl_examples_keyvaluestore_supportexports::{Exportable, Item, StoreMarker}; use zx::Vmo; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; match store.write_item(&Item { key: key, value: value.into_bytes() }).await? { Ok(_) => println!("WriteItem Success"), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // If the `max_export_size` is 0, no export is possible, so just ignore this block. This check // isn't strictly necessary, but does avoid extra work down the line. if config.max_export_size > 0 { // Create a 100Kb VMO to store the resulting export. In a real implementation, we would // likely receive the VMO representing the to-be-written file from file system like vfs of // fxfs. let vmo = Vmo::create(config.max_export_size)?; // Send the VMO to the server, to be populated with the current state of the key-value // store. match store.export(vmo).await? { Err(err) => { println!("Export Error: {}", err.into_primitive()); } Ok(output) => { println!("Export Success"); // Read the exported data (encoded in byte form as persistent FIDL) from the // returned VMO. In a real implementation, instead of reading the VMO, we would // merely forward it to some other storage-handling process. Doing this using a VMO, // rather than FIDL IPC, would save us frivolous reads and writes at each hop. let content_size = output.get_content_size().unwrap(); let mut encoded_bytes = vec![0; content_size as usize]; output.read(&mut encoded_bytes, 0)?; // Decode the persistent FIDL that was just read from the file. let exportable = unpersist::<Exportable>(&encoded_bytes).unwrap(); let items = exportable.items.expect("must always be set"); // Log some information about the exported data. println!("Printing {} exported entries, which are:", items.len()); for item in items.iter() { println!(" * {}", item.key); } } }; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fuchsia_component::server::ServiceFs; use futures::prelude::*; use lazy_static::lazy_static; use regex::Regex; use std::cell::RefCell; use std::collections::hash_map::Entry; use std::collections::HashMap; use fidl::{persist, Vmo}; use fidl_examples_keyvaluestore_supportexports::{ ExportError, Exportable, Item, StoreRequest, StoreRequestStream, WriteError, }; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } /// Handler for the `WriteItem` method. fn write_item(store: &mut HashMap<String, Vec<u8>>, attempt: Item) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Validate the value. if attempt.value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", attempt.key); return Err(WriteError::InvalidValue); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote value at key: {}", entry.key()); entry.insert(attempt.value); Ok(()) } } } /// Handler for the `Export` method. fn export(store: &mut HashMap<String, Vec<u8>>, vmo: Vmo) -> Result<Vmo, ExportError> { // Empty stores cannot be exported. if store.is_empty() { return Err(ExportError::Empty); } // Build the `Exportable` vector locally. That means iterating over the map, and turning it into // a vector of items instead. let mut exportable = Exportable::default(); let mut items = store .iter() .map(|entry| return Item { key: entry.0.clone(), value: entry.1.clone() }) .collect::<Vec<Item>>(); items.sort_by(|a, b| a.key.cmp(&b.key)); exportable.items = Some(items); // Encode the bytes - there is a bug in persistent FIDL if this operation fails. Even if it // succeeds, make sure to check that the VMO has enough space to handle the encoded export data. let encoded_bytes = persist(&exportable).map_err(|_| ExportError::Unknown)?; if encoded_bytes.len() as u64 > vmo.get_content_size().map_err(|_| ExportError::Unknown)? { return Err(ExportError::StorageTooSmall); } // Write the (now encoded) persistent FIDL data to the VMO. vmo.set_content_size(&(encoded_bytes.len() as u64)).map_err(|_| ExportError::Unknown)?; vmo.write(&encoded_bytes, 0).map_err(|_| ExportError::Unknown)?; Ok(vmo) } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, Vec<u8>>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.borrow_mut(), attempt)) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::Export { empty, responder } => { println!("Export request received"); responder .send(export(&mut store.borrow_mut(), empty)) .context("error sending reply")?; println!("Export response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.keyvaluestore.supportexports/cpp/fidl.h> #include <lib/async-loop/cpp/loop.h> #include <lib/component/incoming/cpp/protocol.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <examples/fidl/new/key_value_store/support_exports/cpp_natural/client/config.h> #include <src/lib/files/file.h> #include <src/lib/fxl/strings/string_printf.h> int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop and dispatcher. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace. This can fail so it's wrapped in a // |zx::result| and it must be checked for errors. zx::result client_end = component::Connect<examples_keyvaluestore_supportexports::Store>(); if (!client_end.is_ok()) { FX_LOGS(ERROR) << "Synchronous error when connecting to the |Store| protocol: " << client_end.status_string(); return -1; } // Create an asynchronous client using the newly-established connection. fidl::Client client(std::move(*client_end), dispatcher); FX_LOGS(INFO) << "Outgoing connection enabled"; for (const auto& action : conf.write_items()) { std::string text; if (!files::ReadFileToString(fxl::StringPrintf("/pkg/data/%s.txt", action.c_str()), &text)) { FX_LOGS(ERROR) << "It looks like the correct `resource` dependency has not been packaged"; break; } auto value = std::vector<uint8_t>(text.begin(), text.end()); client->WriteItem(examples_keyvaluestore_supportexports::Item(action, value)) .ThenExactlyOnce( [&](fidl::Result<examples_keyvaluestore_supportexports::Store::WriteItem> result) { // Check if the FIDL call succeeded or not. if (!result.is_ok()) { if (result.error_value().is_framework_error()) { FX_LOGS(ERROR) << "Unexpected FIDL framework error: " << result.error_value(); } else { FX_LOGS(INFO) << "WriteItem Error: " << fidl::ToUnderlying(result.error_value().domain_error()); } } else { FX_LOGS(INFO) << "WriteItem Success"; } // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // If the `max_export_size` is 0, no export is possible, so just ignore this block. This check // isn't strictly necessary, but does avoid extra work down the line. if (conf.max_export_size() > 0) { // Create a 100Kb VMO to store the resulting export. In a real implementation, we would // likely receive the VMO representing the to-be-written file from file system like vfs of // fxfs. zx::vmo vmo; if (zx_status_t status = zx::vmo::create(conf.max_export_size(), 0, &vmo); status != ZX_OK) { FX_PLOGS(ERROR, status) << "Failed to create VMO"; return -1; } client->Export({std::move(vmo)}) .ThenExactlyOnce( [&](fidl::Result<examples_keyvaluestore_supportexports::Store::Export>& result) { // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over, when we return from this callback. loop.Quit(); if (!result.is_ok()) { if (result.error_value().is_framework_error()) { FX_LOGS(ERROR) << "Unexpected FIDL framework error: " << result.error_value(); } else { FX_LOGS(INFO) << "Export Error: " << fidl::ToUnderlying(result.error_value().domain_error()); } return; } FX_LOGS(INFO) << "Export Success"; // Read the exported data (encoded in byte form as persistent FIDL) from the // returned VMO. In a real implementation, instead of reading the VMO, we would // merely forward it to some other storage-handling process. Doing this using a VMO, // rather than FIDL IPC, would save us frivolous reads and writes at each hop. size_t content_size = 0; zx::vmo vmo = std::move(result->filled()); if (vmo.get_prop_content_size(&content_size) != ZX_OK) { return; } std::vector<uint8_t> encoded_bytes; encoded_bytes.resize(content_size); if (vmo.read(encoded_bytes.data(), 0, content_size) != ZX_OK) { return; } // Decode the persistent FIDL that was just read from the file. fit::result exportable = fidl::Unpersist<examples_keyvaluestore_supportexports::Exportable>( cpp20::span(encoded_bytes)); if (exportable.is_error()) { FX_LOGS(ERROR) << "Failed to unpersist: " << exportable.error_value(); return; } if (!exportable->items().has_value()) { FX_LOGS(INFO) << "Expected items to be set"; return; } auto& items = exportable->items().value(); // Log some information about the exported data. FX_LOGS(INFO) << "Printing " << items.size() << " exported entries, which are:"; for (const auto& item : items) { FX_LOGS(INFO) << " * " << item.key(); } }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.keyvaluestore.supportexports/cpp/fidl.h> #include <lib/async-loop/cpp/loop.h> #include <lib/async/cpp/task.h> #include <lib/component/outgoing/cpp/outgoing_directory.h> #include <lib/fidl/cpp/wire/channel.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <algorithm> #include <re2/re2.h> // An implementation of the |Store| protocol. class StoreImpl final : public fidl::Server<examples_keyvaluestore_supportexports::Store> { public: // Bind this implementation to a channel. StoreImpl(async_dispatcher_t* dispatcher, fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) : binding_(fidl::BindServer( dispatcher, std::move(server_end), this, [this](StoreImpl* impl, fidl::UnbindInfo info, fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) { if (info.reason() != ::fidl::Reason::kPeerClosedWhileReading) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; })) {} void WriteItem(WriteItemRequest& request, WriteItemCompleter::Sync& completer) override { FX_LOGS(INFO) << "WriteItem request received"; auto key = request.attempt().key(); auto value = request.attempt().value(); // Validate the key. if (!RE2::FullMatch(key, "^[A-Za-z]\\w+[A-Za-z0-9]$")) { FX_LOGS(INFO) << "Write error: INVALID_KEY, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kInvalidKey)); } // Validate the value. if (value.empty()) { FX_LOGS(INFO) << "Write error: INVALID_VALUE, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kInvalidValue)); } if (key_value_store_.find(key) != key_value_store_.end()) { FX_LOGS(INFO) << "Write error: ALREADY_EXISTS, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kAlreadyExists)); } // Ensure that the value does not already exist in the store. key_value_store_.insert({key, value}); FX_LOGS(INFO) << "Wrote value at key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply(fit::ok()); } void Export(ExportRequest& request, ExportCompleter::Sync& completer) override { FX_LOGS(INFO) << "Export request received"; completer.Reply(Export(std::move(request.empty()))); FX_LOGS(INFO) << "Export response sent"; } void handle_unknown_method( fidl::UnknownMethodMetadata<examples_keyvaluestore_supportexports::Store> metadata, fidl::UnknownMethodCompleter::Sync& completer) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << metadata.method_ordinal; } private: using ExportError = ::examples_keyvaluestore_supportexports::ExportError; using Exportable = ::examples_keyvaluestore_supportexports::Exportable; using Item = ::examples_keyvaluestore_supportexports::Item; fit::result<ExportError, zx::vmo> Export(zx::vmo vmo) { if (key_value_store_.empty()) { return fit::error(ExportError::kEmpty); } Exportable exportable; std::vector<Item> items; items.reserve(key_value_store_.size()); for (const auto& [k, v] : key_value_store_) { items.push_back(Item{{.key = k, .value = v}}); } std::sort(items.begin(), items.end(), [](const Item& a, const Item& b) { return a.key() < b.key(); }); exportable.items(std::move(items)); fit::result encoded = fidl::Persist(exportable); if (encoded.is_error()) { FX_LOGS(ERROR) << "Failed to encode in persistence convention: " << encoded.error_value(); return fit::error(ExportError::kUnknown); } size_t content_size = 0; if (vmo.get_prop_content_size(&content_size) != ZX_OK) { return fit::error(ExportError::kUnknown); } if (encoded->size() > content_size) { return fit::error(ExportError::kStorageTooSmall); } if (vmo.set_prop_content_size(encoded->size()) != ZX_OK) { return fit::error(ExportError::kUnknown); } if (vmo.write(encoded->data(), 0, encoded->size()) != ZX_OK) { return fit::error(ExportError::kUnknown); } return fit::ok(std::move(vmo)); } fidl::ServerBindingRef<examples_keyvaluestore_supportexports::Store> binding_; // The map that serves as the per-connection instance of the key-value store. std::unordered_map<std::string, std::vector<uint8_t>> key_value_store_ = {}; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. This // directory is where the outgoing FIDL protocols are installed so that they can be provided to // other components. component::OutgoingDirectory outgoing = component::OutgoingDirectory(dispatcher); // The `ServeFromStartupInfo()` function sets up the outgoing directory with the startup handle. // The startup handle is a handle provided to every component by the system, so that they can // serve capabilities (e.g. FIDL protocols) to other components. zx::result result = outgoing.ServeFromStartupInfo(); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to serve outgoing directory: " << result.status_string(); return -1; } // Register a handler for components trying to connect to |Store|. result = outgoing.AddUnmanagedProtocol<examples_keyvaluestore_supportexports::Store>( [dispatcher](fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) { // Create an instance of our StoreImpl that destroys itself when the connection closes. new StoreImpl(dispatcher, std::move(server_end)); }); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to add Store protocol: " << result.status_string(); return -1; } // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
C++ (Wire)
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.keyvaluestore.supportexports/cpp/wire.h> #include <lib/async-loop/cpp/loop.h> #include <lib/component/incoming/cpp/protocol.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <examples/fidl/new/key_value_store/support_exports/cpp_wire/client/config.h> #include <src/lib/files/file.h> #include <src/lib/fxl/strings/string_printf.h> int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop and dispatcher. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace. This can fail so it's wrapped in a // |zx::result| and it must be checked for errors. zx::result client_end = component::Connect<examples_keyvaluestore_supportexports::Store>(); if (!client_end.is_ok()) { FX_LOGS(ERROR) << "Synchronous error when connecting to the |Store| protocol: " << client_end.status_string(); return -1; } // Create an asynchronous client using the newly-established connection. fidl::WireClient client(std::move(*client_end), dispatcher); FX_LOGS(INFO) << "Outgoing connection enabled"; for (const auto& key : conf.write_items()) { std::string text; if (!files::ReadFileToString(fxl::StringPrintf("/pkg/data/%s.txt", key.c_str()), &text)) { FX_LOGS(ERROR) << "It looks like the correct `resource` dependency has not been packaged"; break; } auto value = std::vector<uint8_t>(text.begin(), text.end()); client ->WriteItem( {fidl::StringView::FromExternal(key), fidl::VectorView<uint8_t>::FromExternal(value)}) .ThenExactlyOnce( [&](fidl::WireUnownedResult<examples_keyvaluestore_supportexports::Store::WriteItem>& result) { if (!result.ok()) { FX_LOGS(ERROR) << "Unexpected framework error"; } else if (result->is_error()) { FX_LOGS(INFO) << "WriteItem Error: " << fidl::ToUnderlying(result->error_value()); } else { FX_LOGS(INFO) << "WriteItem Success"; } // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // If the `max_export_size` is 0, no export is possible, so just ignore this block. This check // isn't strictly necessary, but does avoid extra work down the line. if (conf.max_export_size() > 0) { // Create a 100Kb VMO to store the resulting export. In a real implementation, we would // likely receive the VMO representing the to-be-written file from file system like vfs of // fxfs. zx::vmo vmo; if (zx_status_t status = zx::vmo::create(conf.max_export_size(), 0, &vmo); status != ZX_OK) { FX_PLOGS(ERROR, status) << "Failed to create VMO"; return -1; } client->Export(std::move(vmo)) .ThenExactlyOnce( [&](fidl::WireUnownedResult<examples_keyvaluestore_supportexports::Store::Export>& result) { // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over, when we return from this callback. loop.Quit(); if (!result.ok()) { FX_LOGS(ERROR) << "Unexpected FIDL framework error: " << result.error(); return; } if (!result->is_ok()) { FX_LOGS(INFO) << "Export Error: " << fidl::ToUnderlying(result->error_value()); return; } FX_LOGS(INFO) << "Export Success"; // Read the exported data (encoded in byte form as persistent FIDL) from the // returned VMO. In a real implementation, instead of reading the VMO, we would // merely forward it to some other storage-handling process. Doing this using a VMO, // rather than FIDL IPC, would save us frivolous reads and writes at each hop. size_t content_size = 0; zx::vmo vmo = std::move(result->value()->filled); if (vmo.get_prop_content_size(&content_size) != ZX_OK) { return; } std::vector<uint8_t> encoded_bytes; encoded_bytes.resize(content_size); if (vmo.read(encoded_bytes.data(), 0, content_size) != ZX_OK) { return; } // Decode the persistent FIDL that was just read from the file. fit::result exportable = fidl::InplaceUnpersist<examples_keyvaluestore_supportexports::wire::Exportable>( cpp20::span(encoded_bytes)); if (exportable.is_error()) { FX_LOGS(ERROR) << "Failed to unpersist: " << exportable.error_value(); return; } if (!exportable->has_items()) { FX_LOGS(INFO) << "Expected items to be set"; return; } auto& items = exportable->items(); // Log some information about the exported data. FX_LOGS(INFO) << "Printing " << items.count() << " exported entries, which are:"; for (const auto& item : items) { FX_LOGS(INFO) << " * " << item.key.get(); } }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.keyvaluestore.supportexports/cpp/wire.h> #include <lib/async-loop/cpp/loop.h> #include <lib/async/cpp/task.h> #include <lib/component/outgoing/cpp/outgoing_directory.h> #include <lib/fidl/cpp/wire/channel.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <algorithm> #include <re2/re2.h> // An implementation of the |Store| protocol. class StoreImpl final : public fidl::WireServer<examples_keyvaluestore_supportexports::Store> { public: // Bind this implementation to a channel. StoreImpl(async_dispatcher_t* dispatcher, fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) : binding_(fidl::BindServer( dispatcher, std::move(server_end), this, [this](StoreImpl* impl, fidl::UnbindInfo info, fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) { if (info.reason() != ::fidl::Reason::kPeerClosedWhileReading) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; })) {} void WriteItem(WriteItemRequestView request, WriteItemCompleter::Sync& completer) override { FX_LOGS(INFO) << "WriteItem request received"; std::string key{request->attempt.key.get()}; std::vector<uint8_t> value{request->attempt.value.begin(), request->attempt.value.end()}; // Validate the key. if (!RE2::FullMatch(key, "^[A-Za-z]\\w+[A-Za-z0-9]$")) { FX_LOGS(INFO) << "Write error: INVALID_KEY, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kInvalidKey)); } // Validate the value. if (value.empty()) { FX_LOGS(INFO) << "Write error: INVALID_VALUE, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kInvalidValue)); } if (key_value_store_.find(key) != key_value_store_.end()) { FX_LOGS(INFO) << "Write error: ALREADY_EXISTS, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kAlreadyExists)); } // Ensure that the value does not already exist in the store. key_value_store_.insert({key, value}); FX_LOGS(INFO) << "Wrote value at key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply(fit::success()); } void Export(ExportRequestView request, ExportCompleter::Sync& completer) override { FX_LOGS(INFO) << "Export request received"; fit::result result = Export(std::move(request->empty)); if (result.is_ok()) { completer.ReplySuccess(std::move(result.value())); } else { completer.ReplyError(result.error_value()); } FX_LOGS(INFO) << "Export response sent"; } using ExportError = ::examples_keyvaluestore_supportexports::wire::ExportError; using Exportable = ::examples_keyvaluestore_supportexports::wire::Exportable; using Item = ::examples_keyvaluestore_supportexports::wire::Item; fit::result<ExportError, zx::vmo> Export(zx::vmo vmo) { if (key_value_store_.empty()) { return fit::error(ExportError::kEmpty); } fidl::Arena arena; fidl::VectorView<Item> items; items.Allocate(arena, key_value_store_.size()); size_t count = 0; for (auto& [k, v] : key_value_store_) { // Create a wire |Item| object that borrows from |k| and |v|. // Since |k| and |v| are references into the long living |key_value_store_|, // while |items| only live within the current function scope, // this operation is safe. items[count] = Item{ .key = fidl::StringView::FromExternal(k), .value = fidl::VectorView<uint8_t>::FromExternal(v), }; count++; } std::sort(items.begin(), items.end(), [](const Item& a, const Item& b) { return a.key.get() < b.key.get(); }); Exportable exportable = Exportable::Builder(arena).items(items).Build(); fit::result encoded = fidl::Persist(exportable); if (encoded.is_error()) { FX_LOGS(ERROR) << "Failed to encode in persistence convention: " << encoded.error_value(); return fit::error(ExportError::kUnknown); } size_t content_size = 0; if (vmo.get_prop_content_size(&content_size) != ZX_OK) { return fit::error(ExportError::kUnknown); } if (encoded->size() > content_size) { return fit::error(ExportError::kStorageTooSmall); } if (vmo.set_prop_content_size(encoded->size()) != ZX_OK) { return fit::error(ExportError::kUnknown); } if (vmo.write(encoded->data(), 0, encoded->size()) != ZX_OK) { return fit::error(ExportError::kUnknown); } return fit::ok(std::move(vmo)); } void handle_unknown_method( fidl::UnknownMethodMetadata<examples_keyvaluestore_supportexports::Store> metadata, fidl::UnknownMethodCompleter::Sync& completer) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << metadata.method_ordinal; } private: fidl::ServerBindingRef<examples_keyvaluestore_supportexports::Store> binding_; // The map that serves as the per-connection instance of the key-value store. // // Out-of-line references in wire types are always mutable. Thus the // |const std::vector<uint8_t>| from the baseline needs to be changed to // non-const as we're making a vector view pointing to it during |Export|, // even though in practice the value is never mutated. std::unordered_map<std::string, std::vector<uint8_t>> key_value_store_ = {}; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. This // directory is where the outgoing FIDL protocols are installed so that they can be provided to // other components. component::OutgoingDirectory outgoing = component::OutgoingDirectory(dispatcher); // The `ServeFromStartupInfo()` function sets up the outgoing directory with the startup handle. // The startup handle is a handle provided to every component by the system, so that they can // serve capabilities (e.g. FIDL protocols) to other components. zx::result result = outgoing.ServeFromStartupInfo(); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to serve outgoing directory: " << result.status_string(); return -1; } // Register a handler for components trying to connect to |Store|. result = outgoing.AddUnmanagedProtocol<examples_keyvaluestore_supportexports::Store>( [dispatcher](fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) { // Create an instance of our StoreImpl that destroys itself when the connection closes. new StoreImpl(dispatcher, std::move(server_end)); }); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to add Store protocol: " << result.status_string(); return -1; } // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Infallible two way method
FIDL recipe: Infallible two way method
An infallible two way method is a FIDL method that cannot return an error value. The only possible failure modes are if the underlying channel has a failure (like failing to connect to one of the endpoints).
In this example, you will create a basic calculator server & client which shows the fundamental setup needed to first define and then serve and consume a FIDL protocol.
First, you will define the interface definitions and test harness. The
interface definition (the .fidl file itself) is the starting point for any new
FIDL protocol. Additionally, the calculator includes the necessary CML and realm
definitions to create a client-server pattern which can be used as project
scaffolding for arbitrary implementations.
See below for the FIDL code:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. // The namespace for this FIDL protocol. This namespace is how both consumers (clients) and providers (servers) reference this protocol. library examples.calculator.baseline; // @discoverable indicates 'Calculator' is a protocol that will be served under the examples.calculator.baseline libarary namespace. https://fuchsia.dev/fuchsia-src/reference/fidl/language/attributes#discoverable . If @discoverable is missing, it will lead to a compile time error when trying to import the library. @discoverable // A limited-functionality calculator 'protocol' that adds and subtracts integers. open protocol Calculator { // Takes as input a struct with two integers, and returns their sum: (a+b)=sum. This method is infallible (no errors can be generated) as two int32's cannot overflow a result type of int64. flexible Add(struct { a int32; b int32; }) -> (struct { sum int64; }); // Takes as input a struct with two integers, and returns their difference: (a-b)=difference. This method is infallible (no errors can be generated) as two int32's cannot overflow a result type of int64. flexible Subtract(struct { a int32; b int32; }) -> (struct { difference int64; }); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.calculator.baseline.Calculator" }, ], config: {}, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.calculator.baseline.Calculator" }, ], expose: [ { protocol: "examples.calculator.baseline.Calculator", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.calculator.baseline.Calculator", from: "#server", to: "#client", }, // Route logging support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// TODO(https://fxbug.dev/42063075): Rust implementation.Server
// TODO(https://fxbug.dev/42063075): Rust implementation.C++ (Natural)
Client
// TODO(https://fxbug.dev/42063075): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42063075): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42063075): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42063075): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42063075): HLCPP implementation.Server
// TODO(https://fxbug.dev/42063075): HLCPP implementation.Creating a FIDL protocol from the ground up as is shown in this example can be a more common scenario for certain developers, such as platform developers. However, other types of developers also benefit from learning how to construct a FIDL protocol even if they won't typically do so. This helps you learn how everything about FIDL fits together, including the syntax, grammar, language features, how to serve and consume a given FIDL protocol, and how the build system works. For next steps, the examples which follow this baseline show how to extend an existing FIDL protocol, which is expected to be a fairly common practice.
Named payload
FIDL recipe: Named payload
Named payloads are struct, table, or union types that are used directly
as method request or response payloads. These can be used in cases where a
method payload would otherwise be repeated or is already a named type used
elsewhere in the FIDL file.
In the following example, the added ReadItem method is notably different from
the existing WriteItem, using an already existing named type as the payload,
rather than a repetitive inline definition.
Reasoning
The original write-only key-value store is now extended with the ability to read items back out of the store.
Implementation
The changes applied to the FIDL and CML definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.addreaditem; // Aliases for the key and value. Using aliases helps increase the readability of FIDL files and // reduces likelihood of errors due to differing constraints. alias Key = string:128; alias Value = vector<byte>:64000; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key Key; value Value; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// An enumeration of things that may go wrong when trying to read a value out of our store. type ReadError = flexible enum { UNKNOWN = 0; NOT_FOUND = 1; }; /// A very basic key-value store - so basic, in fact, that one may only write to it, never read! @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; /// Reads an item from the store. flexible ReadItem(struct { key Key; }) -> (Item) error ReadError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.addreaditem.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, read_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.addreaditem.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.addreaditem.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.addreaditem.Store", from: "#server", to: "#client", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations for all languages change as well:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use config::Config; use fidl_examples_keyvaluestore_addreaditem::{Item, StoreMarker}; use fuchsia_component::client::connect_to_protocol; use std::{str, thread, time}; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; match store.write_item(&Item { key: key, value: value.into_bytes() }).await? { Ok(_) => println!("WriteItem Success"), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // The structured config for this client contains `read_items`, a vector of strings, each of // which is meant to be read from the key-value store. We iterate over these keys, attempting to // read them in turn. for key in config.read_items.into_iter() { let res = store.read_item(key.as_str()).await; match res.unwrap() { Ok(val) => { println!("ReadItem Success: key: {}, value: {}", key, str::from_utf8(&val.1)?) } Err(err) => println!("ReadItem Error: {}", err.into_primitive()), } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, fidl_examples_keyvaluestore_addreaditem::{ Item, ReadError, StoreRequest, StoreRequestStream, WriteError, }, fuchsia_component::server::ServiceFs, futures::prelude::*, lazy_static::lazy_static, regex::Regex, std::cell::RefCell, std::collections::hash_map::Entry, std::collections::HashMap, }; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z][A-Za-z0-9_\./]{2,62}[A-Za-z0-9]$") .expect("Key validation regex failed to compile"); } /// Handler for the `WriteItem` method. fn write_item(store: &mut HashMap<String, Vec<u8>>, attempt: Item) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Validate the value. if attempt.value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", attempt.key); return Err(WriteError::InvalidValue); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote value at key: {}", entry.key()); entry.insert(attempt.value); Ok(()) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, Vec<u8>>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.borrow_mut(), attempt)) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::ReadItem { key, responder } => { println!("ReadItem request received"); // Read the item from the store, returning the appropriate error if it could not be found. responder .send(match store.borrow().get(&key) { Some(found) => { println!("Read value at key: {}", key); Ok((&key, found)) } None => { println!("Read error: NOT_FOUND, For key: {}", key); Err(ReadError::NotFound) } }) .context("error sending reply")?; println!("ReadItem response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Optionality
FIDL recipe: Optionality
Certain FIDL types can be made optional with no change to the wire shape of
their containing message with the addition of the :optional constraint.
Further, the table layout is always optional, while the struct layout never
is. To make a struct optional, it must be wrapped in a box<T>, thereby
changing the wire shape of its containing message.
| Base type | Optional version | Does optionality change the wire layout? |
|---|---|---|
struct {...} |
box<struct {...}> |
Yes |
table {...} |
table {...} |
No |
union {...} |
union {...}:optional |
No |
vector<T> |
vector<T>:optional |
No |
string |
string:optional |
No |
zx.Handle |
zx.Handle:optional |
No |
client_end:P |
client_end:<P, optional> |
No |
server_end:P |
server_end:<P, optional> |
No |
All other types (bits, enum, array<T, N>, and the primitive types) cannot
be made optional.
In this variant, we allow our key-value store to take other key-value stores as
members. In short, we turn it into a tree. We do this by replacing the original
definition of value with one that utilizes a two-member union: one variant
stores leaf nodes using the same vector<byte> type as before, while the other
stores branch nodes in the form of other nested stores.
Reasoning
Here, we see several uses of optionality, whereby we can declare a type that may or may not exist. There are three flavors of optionality in FIDL:
- Types that have are always stored
out-of-line
on the wire, and thus have a builtin way to describe "absentness" via the
null envelope. Enabling
optionality for these types doesn't affect the wire shape of messages they are
included in - it simply changes which values are valid for that particular
type. The
union,vector<T>,client_end,server_end, andzx.Handletypes can all be made optional via the addition of the:optionalconstraint. By making ourvalueunionoptional, we are able to introduce a canonical "null" entry, in the form of an absentvalue. This means that emptybytesand absent/emptystoreproperties are invalid values. - Unlike the aforementioned types, the
structlayout has no extra space where a null header can be stored. Because of this, it needs to be wrapped in an envelope, changing the on-the-wire shape of the message it is being included in. To ensure that this wire-modifying effect easily legible, theItemstructtype must be wrapped in abox<T>type template. - Finally,
tablelayouts are always optional. An absenttableis simply one with none of its members set.
Trees are a naturally self-referential data structure: any node in the tree may
contain a leaf with pure data (in this case, a string), or a sub-tree with more
nodes. This requires recursion: the definition of Item is now transitively
dependent on itself! Representing recursive types in FIDL can be a bit tricky,
especially because support is currently somewhat
limited. We can support such types as long as there is
at least one optional type in the cycle created by the self-reference. For
instance, here we define the items struct member to be a box<Item>,
thereby breaking the includes cycle.
These changes also make heavy use of anonymous types, or types whose
declarations are inlined at their sole point of use, rather than being named,
top-level type declarations of their own. By default, the names of anonymous
types in the generated language bindings are taken from their local context. For
instance, the newly introduced flexible union takes on its owning member's
name Value, the newly introduced struct would become Store, and so on.
Because this heuristic can sometimes cause collisions, FIDL provides an escape
hatch by allowing the author to manually override an anonymous type's generated
name. This is done via the @generated_name attribute, which allows one to
change the name generated by backends. We can use one here, where the would-be
Store type is renamed to NestedStore to prevent a name collision with the
protocol declaration that uses that same name.
Implementation
The FIDL, CML, and realm interface definitions are modified as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.supporttrees; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value strict union { // Keep the original `bytes` as one of the options in the new union. All leaf nodes in the // tree must be `bytes`, or absent unions (representing empty). Empty byte arrays are // disallowed. 1: bytes vector<byte>:64000; // Allows a store within a store, thereby turning our flat key-value store into a tree // thereof. Note the use of `@generated_name` to prevent a type-name collision with the // `Store` protocol below, and the use of `box<T>` to ensure that there is a break in the // chain of recursion, thereby allowing `Item` to include itself in its own definition. // // This is a table so that added fields, like for example a `hash`, can be easily added in // the future. 2: store @generated_name("nested_store") table { 1: items vector<box<Item>>; }; }:optional; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// A very basic key-value store. @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.supporttrees.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A newline separated list nested entries. The first line should be the key // for the nested store, and each subsequent entry should be a pointer to a text file // containing the string value. The name of that text file (without the `.txt` suffix) will // serve as the entries key. write_nested: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A list of keys, all of which will be populated as null entries. write_null: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.supporttrees.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.supporttrees.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.supporttrees.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, config::Config, fidl_examples_keyvaluestore_supporttrees::{Item, NestedStore, StoreMarker, Value}, fuchsia_component::client::connect_to_protocol, std::{thread, time}, }; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; let res = store .write_item(&Item { key: key.clone(), value: Some(Box::new(Value::Bytes(value.into_bytes()))), }) .await; match res? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // Add nested entries to the key-value store as well. The entries are strings, where the first // line is the key of the entry, and each subsequent entry should be a pointer to a text file // containing the string value. The name of that text file (without the `.txt` suffix) will // serve as the entries key. for spec in config.write_nested.into_iter() { let mut items = vec![]; let mut nested_store = NestedStore::default(); let mut lines = spec.split("\n"); let key = lines.next().unwrap(); // For each entry, make a new entry in the `NestedStore` being built. for entry in lines { let path = format!("/pkg/data/{}.txt", entry); let contents = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; items.push(Some(Box::new(Item { key: entry.to_string(), value: Some(Box::new(Value::Bytes(contents.into()))), }))); } nested_store.items = Some(items); // Send the `NestedStore`, represented as a vector of values. let res = store .write_item(&Item { key: key.to_string(), value: Some(Box::new(Value::Store(nested_store))), }) .await; match res? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // Each entry in this list is a null value in the store. for key in config.write_null.into_iter() { match store.write_item(&Item { key: key.to_string(), value: None }).await? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. // Note: For the clarity of this example, allow code to be unused. #![allow(dead_code)] use { anyhow::{Context as _, Error}, fidl_examples_keyvaluestore_supporttrees::{ Item, StoreRequest, StoreRequestStream, Value, WriteError, }, fuchsia_component::server::ServiceFs, futures::prelude::*, lazy_static::lazy_static, regex::Regex, std::cell::RefCell, std::collections::hash_map::Entry, std::collections::HashMap, std::str::from_utf8, }; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } // A representation of a key-value store that can contain an arbitrarily deep nesting of other // key-value stores. #[allow(clippy::box_collection, reason = "mass allow for https://fxbug.dev/381896734")] enum StoreNode { Leaf(Option<Vec<u8>>), Branch(Box<HashMap<String, StoreNode>>), } /// Recursive item writer, which takes a `StoreNode` that may not necessarily be the root node, and /// writes an entry to it. fn write_item( store: &mut HashMap<String, StoreNode>, attempt: Item, path: &str, ) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { let key = format!("{}{}", &path, entry.key()); match attempt.value { // Null entries are allowed. None => { println!("Wrote value: NONE at key: {}", key); entry.insert(StoreNode::Leaf(None)); } Some(value) => match *value { // If this is a nested store, recursively make a new store to insert at this // position. Value::Store(entry_list) => { // Validate the value - absent stores, items lists with no children, or any // of the elements within that list being empty boxes, are all not allowed. if entry_list.items.is_some() { let items = entry_list.items.unwrap(); if !items.is_empty() && items.iter().all(|i| i.is_some()) { let nested_path = format!("{}/", key); let mut nested_store = HashMap::<String, StoreNode>::new(); for item in items.into_iter() { write_item(&mut nested_store, *item.unwrap(), &nested_path)?; } println!("Created branch at key: {}", key); entry.insert(StoreNode::Branch(Box::new(nested_store))); return Ok(()); } } println!("Write error: INVALID_VALUE, For key: {}", key); return Err(WriteError::InvalidValue); } // This is a simple leaf node on this branch. Value::Bytes(value) => { // Validate the value. if value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", key); return Err(WriteError::InvalidValue); } println!("Wrote key: {}, value: {:?}", key, from_utf8(&value).unwrap()); entry.insert(StoreNode::Leaf(Some(value))); } }, } Ok(()) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, StoreNode>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.borrow_mut(), attempt, "")) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Pagination pattern
FIDL recipe: Pagination pattern
When sending a list of items that can potentially get very large, a useful strategy is to break that list up over multiple calls via the pagination pattern. Using pagination allows for more granular synchronization of work between the sender and receiver: instead of pummeling the receiving party with a massive list, the sender emits a few items at a time and waits for feedback that the message has been processed before continuing.
In FIDL terms, this means that rather than sending a single large vector<T>,
FIDL authors should instead convert it to an acked message of vector<T>:N to
ensure that the page size and flow control is part of the FIDL contract.
A useful operation for key-value stores is in-order iteration: that is, when given a key, to return a (usually paginated) list of elements that appear after it, in order.
Reasoning
In FIDL, this is best done using an iterator, which is generally implemented as a separate protocol over which this iteration can occur. Using a separate protocol, and therefore a separate channel, has a number of benefits, including de-interleaving the iteration pull requests from other operations done over the main protocol.
The client and server side of the channel connection for protocol P can be
represented as FIDL data types, as a client_end:P and server_end:P,
respectively. These types are collectively known as protocol ends, and
represent the other (non-@discoverable) way of connecting a FIDL client to its
corresponding server: over an existing FIDL connection!
Protocol ends are specific instances of a general FIDL concept: the resource type. A resource type is intended to contain FIDL handles, which necessitates extra restrictions on how the type can be used. The type must be always be unique, as the underlying resource is mediated by some other capability manager (usually the Zircon kernel). Duplicating such a resource via a simple in-memory copy, without involving the manager, is impossible. To prevent such duplication, all resource types in FIDL are always move-only.
Finally, the Get() method of the Iterator protocol itself makes use of a
size constraint on the return payload. This limits the amount of data that may
be transmitted in a single pull, allowing for some measure of resource use
control. It also creates a natural pagination boundary: rather than a giant dump
of all of the results at once, the server only needs to prepare small batches at
a time.
Implementation
The FIDL, CML, and realm interface definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.additerator; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value vector<byte>:64000; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// An enumeration of things that may go wrong when trying to create an iterator. type IterateConnectionError = flexible enum { /// The starting key was not found. UNKNOWN_START_AT = 1; }; /// A key-value store which supports insertion and iteration. @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; /// Iterates over the items in the store, using lexicographic ordering over the keys. /// /// The [`iterator`] is [pipelined][pipelining] to the server, such that the client can /// immediately send requests over the new connection. /// /// [pipelining]: https://fuchsia.dev/fuchsia-src/development/api/fidl?hl=en#request-pipelining flexible Iterate(resource struct { /// If present, requests to start the iteration at this item. starting_at string:<128, optional>; /// The [`Iterator`] server endpoint. The client creates both ends of the channel and /// retains the `client_end` locally to use for pulling iteration pages, while sending the /// `server_end` off to be fulfilled by the server. iterator server_end:Iterator; }) -> () error IterateConnectionError; }; /// An iterator for the key-value store. Note that this protocol makes no guarantee of atomicity - /// the values may change between pulls from the iterator. Unlike the `Store` protocol above, this /// protocol is not `@discoverable`: it is not independently published by the component that /// implements it, but rather must have one of its two protocol ends transmitted over an existing /// FIDL connection. /// /// As is often the case with iterators, the client indicates that they are done with an instance of /// the iterator by simply closing their end of the connection. /// /// Since the iterator is associated only with the Iterate method, it is declared as closed rather /// than open. This is because changes to how iteration works are more likely to require replacing /// the Iterate method completely (which is fine because that method is flexible) rather than /// evolving the Iterator protocol. closed protocol Iterator { /// Gets the next batch of keys. /// /// The client pulls keys rather than having the server proactively push them, to implement /// [flow control][flow-control] over the messages. /// /// [flow-control]: /// https://fuchsia.dev/fuchsia-src/development/api/fidl?hl=en#prefer_pull_to_push strict Get() -> (struct { /// A list of keys. If the iterator has reached the end of iteration, the list will be /// empty. The client is expected to then close the connection. entries vector<string:128>:10; }); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.additerator.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A key to iterate from, after all items in `write_items` have been written. iterate_from: { type: "string", max_size: 64, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.additerator.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.additerator.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.additerator.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use config::Config; use fuchsia_component::client::connect_to_protocol; use std::{thread, time}; use fidl::endpoints::create_proxy; use fidl_examples_keyvaluestore_additerator::{Item, IteratorMarker, StoreMarker}; use futures::join; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; match store.write_item(&Item { key: key, value: value.into_bytes() }).await? { Ok(_) => println!("WriteItem Success"), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } if !config.iterate_from.is_empty() { // This helper creates a channel, and returns two protocol ends: the `client_end` is already // conveniently bound to the correct FIDL protocol, `Iterator`, while the `server_end` is // unbound and ready to be sent over the wire. let (iterator, server_end) = create_proxy::<IteratorMarker>(); // There is no need to wait for the iterator to connect before sending the first `Get()` // request - since we already hold the `client_end` of the connection, we can start queuing // requests on it immediately. let connect_to_iterator = store.iterate(Some(config.iterate_from.as_str()), server_end); let first_get = iterator.get(); // Wait until both the connection and the first request resolve - an error in either case // triggers an immediate resolution of the combined future. let (connection, first_page) = join!(connect_to_iterator, first_get); // Handle any connection error. If this has occurred, it is impossible for the first `Get()` // call to have resolved successfully, so check this error first. if let Err(err) = connection.context("Could not connect to Iterator")? { println!("Iterator Connection Error: {}", err.into_primitive()); } else { println!("Iterator Connection Success"); // Consecutively repeat the `Get()` request if the previous response was not empty. let mut entries = first_page.context("Could not get page from Iterator")?; while !&entries.is_empty() { for entry in entries.iter() { println!("Iterator Entry: {}", entry); } entries = iterator.get().await.context("Could not get page from Iterator")?; } } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fuchsia_component::server::ServiceFs; use futures::prelude::*; use lazy_static::lazy_static; use regex::Regex; use fidl_examples_keyvaluestore_additerator::{ Item, IterateConnectionError, IteratorRequest, IteratorRequestStream, StoreRequest, StoreRequestStream, WriteError, }; use fuchsia_async as fasync; use std::collections::btree_map::Entry; use std::collections::BTreeMap; use std::ops::Bound::*; use std::sync::{Arc, Mutex}; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } /// Handler for the `WriteItem` method. fn write_item(store: &mut BTreeMap<String, Vec<u8>>, attempt: Item) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Validate the value. if attempt.value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", attempt.key); return Err(WriteError::InvalidValue); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote value at key: {}", entry.key()); entry.insert(attempt.value); Ok(()) } } } /// Handler for the `Iterate` method, which deals with validating that the requested start position /// exists, and then sets up the asynchronous side channel for the actual iteration to occur over. fn iterate( store: Arc<Mutex<BTreeMap<String, Vec<u8>>>>, starting_at: Option<String>, stream: IteratorRequestStream, ) -> Result<(), IterateConnectionError> { // Validate that the starting key, if supplied, actually exists. if let Some(start_key) = starting_at.clone() { if !store.lock().unwrap().contains_key(&start_key) { return Err(IterateConnectionError::UnknownStartAt); } } // Spawn a detached task. This allows the method call to return while the iteration continues in // a separate, unawaited task. fasync::Task::spawn(async move { // Serve the iteration requests. Note that access to the underlying store is behind a // contended `Mutex`, meaning that the iteration is not atomic: page contents could shift, // change, or disappear entirely between `Get()` requests. stream .map(|result| result.context("failed request")) .try_fold( match starting_at { Some(start_key) => Included(start_key), None => Unbounded, }, |mut lower_bound, request| async { match request { IteratorRequest::Get { responder } => { println!("Iterator page request received"); // The `page_size` should be kept in sync with the size constraint on // the iterator's response, as defined in the FIDL protocol. static PAGE_SIZE: usize = 10; // An iterator, beginning at `lower_bound` and tracking the pagination's // progress through iteration as each page is pulled by a client-sent // `Get()` request. let held_store = store.lock().unwrap(); let mut entries = held_store.range((lower_bound.clone(), Unbounded)); let mut current_page = vec![]; for _ in 0..PAGE_SIZE { match entries.next() { Some(entry) => { current_page.push(entry.0.clone()); } None => break, } } // Update the `lower_bound` - either inclusive of the next item in the // iteration, or exclusive of the last seen item if the iteration has // finished. This `lower_bound` will be passed to the next request // handler as its starting point. lower_bound = match entries.next() { Some(next) => Included(next.0.clone()), None => match current_page.last() { Some(tail) => Excluded(tail.clone()), None => lower_bound, }, }; // Send the page. At the end of this scope, the `held_store` lock gets // dropped, and therefore released. responder.send(¤t_page).context("error sending reply")?; println!("Iterator page sent"); } } Ok(lower_bound) }, ) .await .ok(); }) .detach(); Ok(()) } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. // // Note that we now use an `Arc<Mutex<BTreeMap>>`, replacing the previous `RefCell<HashMap>`. // The `BTreeMap` is used because we want an ordered map, to better facilitate iteration. The // `Arc<Mutex<...>>` is used because there are now multiple async tasks accessing the: one main // task which handles communication over the protocol, and one additional task per iterator // protocol. `Arc<Mutex<...>>` is the simplest way to synchronize concurrent access between // these racing tasks. let store = &Arc::new(Mutex::new(BTreeMap::<String, Vec<u8>>::new())); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.clone().lock().unwrap(), attempt)) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::Iterate { starting_at, iterator, responder } => { println!("Iterate request received"); // The `iterate` handler does a quick check to see that the request is valid, // then spins up a separate worker task to serve the newly minted `Iterator` // protocol instance, allowing this call to return immediately and continue the // request stream with other work. responder .send(iterate(store.clone(), starting_at, iterator.into_stream())) .context("error sending reply")?; println!("Iterate response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Persistence
FIDL recipe: Persistence
Persistent FIDL refers to wire-encoded binary FIDL data that is stored with no underlying transport. Instead, the data is stored for some arbitrarily long period of time using a persistent, byte-oriented interface like a file or database entry.
A simple way to extend the key-value store to support exporting backups would be
to simply add a new method that stops the world, serializes the state of the
store, and sends it back as a FIDL vector<Item>. There are two downsides to
this approach, however. The first is that it puts all of the burden of the
backup on the server - a client pays nothing to ask for a backup operation that
is very expensive to the server. The second is that it involves a great deal of
copying: the client is almost certainly just going to write the resulting backup
to some backing datastore, like a file or a database, as soon as it receives it.
Having it decode this (potentially very large) FIDL object, just so that it can
immediately re-encode it as it forwards it to whatever protocol will do the
actual storage, is very wasteful.
Reasoning
A better solution is to use zircon's virtual memory objects. Instead of constantly copying bytes back and forth in a bucket brigade, we can mint a VMO to hold the backup data on the client, send it to the server, then forward it back to our target data store without deserializing in between. As long as the target data store's protocol has allowances for accepting data transported using a VMO, this is the preferred way to accomplish expensive operations like this. In fact, Fuchsia's file system, for instance, implements this exact pattern. A benefit of this approach is that it forces the client to do some work when asking the server for an expensive operation, minimizing the work imbalance between the two parties.
FIDL value types can be persisted to any byte-oriented storage medium, using
the FIDL data persistence binary format. We will persist the
newly introduced FIDL type Exportable into the VMO. The object will be encoded
and written to the storage (in this case, a VMO that could later be saved as a
file), and decoded from it when the data needs to be accessed again, in much the
same way that a message is encoded, transported, and decoded again later when
using FIDL over IPC.
To do this securely and adhere to the principle of least privilege,
we should constrain the privileges the handle representing our VMO may carry.
Enter handle rights, FIDL's first-class method of describing the privileges
available to a particular handle type. In this case, we allow the empty VMO
passed to the server in the Export request to be read from, queried for size,
resized, and written to. When the VMO is returned, we remove right to resize and
write, ensuring that no process, not even malicious actors in some far away
component, can modify this data as it moves through the system.
Implementation
The FIDL, CML, and realm interface definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.supportexports; using zx; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value vector<byte>:64000; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// An enumeration of things that may go wrong when trying to mint an export. type ExportError = flexible enum { UNKNOWN = 0; EMPTY = 1; STORAGE_TOO_SMALL = 2; }; // A data type describing the structure of a single export. We never actually send this data type // over the wire (we use the file's VMO instead), but whenever data needs to be written to/read from // its backing storage as persistent FIDL, it will have this schema. /// /// The items should be sorted in ascending order, following lexicographic ordering of their keys. type Exportable = table { 1: items vector<Item>; }; /// A very basic key-value store - so basic, in fact, that one may only write to it, never read! @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; /// Exports the entire store as a persistent [`Exportable`] FIDL object into a VMO provided by /// the client. /// /// By having the client provide (and speculatively size) the VMO, we force the party requesting /// the relatively heavy load of generating a backup to acknowledge and bear some of the costs. /// /// This method operates by having the client supply an empty VMO, which the server then /// attempts to fill. Notice that the server removes the `zx.Rights.WRITE` and /// `zx.Rights.SET_PROPERTY` rights from the returned VMO - not even the requesting client may /// alter the backup once it has been minted by the server. flexible Export(resource struct { /// Note that the empty VMO has more rights than the filled one being returned: it has /// `zx.Rights.WRITE` (via `zx.RIGHTS_IO`) so that the VMO may be filled with exported data, /// and `zx.Rights.SET_PROPERTY` (via `zx.RIGHTS_PROPERTY`) so that it may be resized to /// truncate any remaining empty buffer. empty zx.Handle:<VMO, zx.RIGHTS_BASIC | zx.RIGHTS_PROPERTY | zx.RIGHTS_IO>; }) -> (resource struct { /// The `zx.Rights.WRITE` and `zx.Rights.SET_PROPERTY` rights have been removed from the now /// filled VMO. No one, not even the client that requested the export, is able to modify /// this VMO going forward. filled zx.Handle:<VMO, zx.RIGHTS_BASIC | zx.Rights.GET_PROPERTY | zx.Rights.READ>; }) error ExportError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.supportexports.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // The size, in bytes, allotted to the export VMO max_export_size: { type: "uint64" }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.supportexports.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.supportexports.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.supportexports.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use config::Config; use fuchsia_component::client::connect_to_protocol; use std::{thread, time}; use fidl::unpersist; use fidl_examples_keyvaluestore_supportexports::{Exportable, Item, StoreMarker}; use zx::Vmo; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; match store.write_item(&Item { key: key, value: value.into_bytes() }).await? { Ok(_) => println!("WriteItem Success"), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // If the `max_export_size` is 0, no export is possible, so just ignore this block. This check // isn't strictly necessary, but does avoid extra work down the line. if config.max_export_size > 0 { // Create a 100Kb VMO to store the resulting export. In a real implementation, we would // likely receive the VMO representing the to-be-written file from file system like vfs of // fxfs. let vmo = Vmo::create(config.max_export_size)?; // Send the VMO to the server, to be populated with the current state of the key-value // store. match store.export(vmo).await? { Err(err) => { println!("Export Error: {}", err.into_primitive()); } Ok(output) => { println!("Export Success"); // Read the exported data (encoded in byte form as persistent FIDL) from the // returned VMO. In a real implementation, instead of reading the VMO, we would // merely forward it to some other storage-handling process. Doing this using a VMO, // rather than FIDL IPC, would save us frivolous reads and writes at each hop. let content_size = output.get_content_size().unwrap(); let mut encoded_bytes = vec![0; content_size as usize]; output.read(&mut encoded_bytes, 0)?; // Decode the persistent FIDL that was just read from the file. let exportable = unpersist::<Exportable>(&encoded_bytes).unwrap(); let items = exportable.items.expect("must always be set"); // Log some information about the exported data. println!("Printing {} exported entries, which are:", items.len()); for item in items.iter() { println!(" * {}", item.key); } } }; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fuchsia_component::server::ServiceFs; use futures::prelude::*; use lazy_static::lazy_static; use regex::Regex; use std::cell::RefCell; use std::collections::hash_map::Entry; use std::collections::HashMap; use fidl::{persist, Vmo}; use fidl_examples_keyvaluestore_supportexports::{ ExportError, Exportable, Item, StoreRequest, StoreRequestStream, WriteError, }; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } /// Handler for the `WriteItem` method. fn write_item(store: &mut HashMap<String, Vec<u8>>, attempt: Item) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Validate the value. if attempt.value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", attempt.key); return Err(WriteError::InvalidValue); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote value at key: {}", entry.key()); entry.insert(attempt.value); Ok(()) } } } /// Handler for the `Export` method. fn export(store: &mut HashMap<String, Vec<u8>>, vmo: Vmo) -> Result<Vmo, ExportError> { // Empty stores cannot be exported. if store.is_empty() { return Err(ExportError::Empty); } // Build the `Exportable` vector locally. That means iterating over the map, and turning it into // a vector of items instead. let mut exportable = Exportable::default(); let mut items = store .iter() .map(|entry| return Item { key: entry.0.clone(), value: entry.1.clone() }) .collect::<Vec<Item>>(); items.sort_by(|a, b| a.key.cmp(&b.key)); exportable.items = Some(items); // Encode the bytes - there is a bug in persistent FIDL if this operation fails. Even if it // succeeds, make sure to check that the VMO has enough space to handle the encoded export data. let encoded_bytes = persist(&exportable).map_err(|_| ExportError::Unknown)?; if encoded_bytes.len() as u64 > vmo.get_content_size().map_err(|_| ExportError::Unknown)? { return Err(ExportError::StorageTooSmall); } // Write the (now encoded) persistent FIDL data to the VMO. vmo.set_content_size(&(encoded_bytes.len() as u64)).map_err(|_| ExportError::Unknown)?; vmo.write(&encoded_bytes, 0).map_err(|_| ExportError::Unknown)?; Ok(vmo) } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, Vec<u8>>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.borrow_mut(), attempt)) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::Export { empty, responder } => { println!("Export request received"); responder .send(export(&mut store.borrow_mut(), empty)) .context("error sending reply")?; println!("Export response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.keyvaluestore.supportexports/cpp/fidl.h> #include <lib/async-loop/cpp/loop.h> #include <lib/component/incoming/cpp/protocol.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <examples/fidl/new/key_value_store/support_exports/cpp_natural/client/config.h> #include <src/lib/files/file.h> #include <src/lib/fxl/strings/string_printf.h> int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop and dispatcher. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace. This can fail so it's wrapped in a // |zx::result| and it must be checked for errors. zx::result client_end = component::Connect<examples_keyvaluestore_supportexports::Store>(); if (!client_end.is_ok()) { FX_LOGS(ERROR) << "Synchronous error when connecting to the |Store| protocol: " << client_end.status_string(); return -1; } // Create an asynchronous client using the newly-established connection. fidl::Client client(std::move(*client_end), dispatcher); FX_LOGS(INFO) << "Outgoing connection enabled"; for (const auto& action : conf.write_items()) { std::string text; if (!files::ReadFileToString(fxl::StringPrintf("/pkg/data/%s.txt", action.c_str()), &text)) { FX_LOGS(ERROR) << "It looks like the correct `resource` dependency has not been packaged"; break; } auto value = std::vector<uint8_t>(text.begin(), text.end()); client->WriteItem(examples_keyvaluestore_supportexports::Item(action, value)) .ThenExactlyOnce( [&](fidl::Result<examples_keyvaluestore_supportexports::Store::WriteItem> result) { // Check if the FIDL call succeeded or not. if (!result.is_ok()) { if (result.error_value().is_framework_error()) { FX_LOGS(ERROR) << "Unexpected FIDL framework error: " << result.error_value(); } else { FX_LOGS(INFO) << "WriteItem Error: " << fidl::ToUnderlying(result.error_value().domain_error()); } } else { FX_LOGS(INFO) << "WriteItem Success"; } // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // If the `max_export_size` is 0, no export is possible, so just ignore this block. This check // isn't strictly necessary, but does avoid extra work down the line. if (conf.max_export_size() > 0) { // Create a 100Kb VMO to store the resulting export. In a real implementation, we would // likely receive the VMO representing the to-be-written file from file system like vfs of // fxfs. zx::vmo vmo; if (zx_status_t status = zx::vmo::create(conf.max_export_size(), 0, &vmo); status != ZX_OK) { FX_PLOGS(ERROR, status) << "Failed to create VMO"; return -1; } client->Export({std::move(vmo)}) .ThenExactlyOnce( [&](fidl::Result<examples_keyvaluestore_supportexports::Store::Export>& result) { // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over, when we return from this callback. loop.Quit(); if (!result.is_ok()) { if (result.error_value().is_framework_error()) { FX_LOGS(ERROR) << "Unexpected FIDL framework error: " << result.error_value(); } else { FX_LOGS(INFO) << "Export Error: " << fidl::ToUnderlying(result.error_value().domain_error()); } return; } FX_LOGS(INFO) << "Export Success"; // Read the exported data (encoded in byte form as persistent FIDL) from the // returned VMO. In a real implementation, instead of reading the VMO, we would // merely forward it to some other storage-handling process. Doing this using a VMO, // rather than FIDL IPC, would save us frivolous reads and writes at each hop. size_t content_size = 0; zx::vmo vmo = std::move(result->filled()); if (vmo.get_prop_content_size(&content_size) != ZX_OK) { return; } std::vector<uint8_t> encoded_bytes; encoded_bytes.resize(content_size); if (vmo.read(encoded_bytes.data(), 0, content_size) != ZX_OK) { return; } // Decode the persistent FIDL that was just read from the file. fit::result exportable = fidl::Unpersist<examples_keyvaluestore_supportexports::Exportable>( cpp20::span(encoded_bytes)); if (exportable.is_error()) { FX_LOGS(ERROR) << "Failed to unpersist: " << exportable.error_value(); return; } if (!exportable->items().has_value()) { FX_LOGS(INFO) << "Expected items to be set"; return; } auto& items = exportable->items().value(); // Log some information about the exported data. FX_LOGS(INFO) << "Printing " << items.size() << " exported entries, which are:"; for (const auto& item : items) { FX_LOGS(INFO) << " * " << item.key(); } }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.keyvaluestore.supportexports/cpp/fidl.h> #include <lib/async-loop/cpp/loop.h> #include <lib/async/cpp/task.h> #include <lib/component/outgoing/cpp/outgoing_directory.h> #include <lib/fidl/cpp/wire/channel.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <algorithm> #include <re2/re2.h> // An implementation of the |Store| protocol. class StoreImpl final : public fidl::Server<examples_keyvaluestore_supportexports::Store> { public: // Bind this implementation to a channel. StoreImpl(async_dispatcher_t* dispatcher, fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) : binding_(fidl::BindServer( dispatcher, std::move(server_end), this, [this](StoreImpl* impl, fidl::UnbindInfo info, fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) { if (info.reason() != ::fidl::Reason::kPeerClosedWhileReading) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; })) {} void WriteItem(WriteItemRequest& request, WriteItemCompleter::Sync& completer) override { FX_LOGS(INFO) << "WriteItem request received"; auto key = request.attempt().key(); auto value = request.attempt().value(); // Validate the key. if (!RE2::FullMatch(key, "^[A-Za-z]\\w+[A-Za-z0-9]$")) { FX_LOGS(INFO) << "Write error: INVALID_KEY, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kInvalidKey)); } // Validate the value. if (value.empty()) { FX_LOGS(INFO) << "Write error: INVALID_VALUE, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kInvalidValue)); } if (key_value_store_.find(key) != key_value_store_.end()) { FX_LOGS(INFO) << "Write error: ALREADY_EXISTS, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kAlreadyExists)); } // Ensure that the value does not already exist in the store. key_value_store_.insert({key, value}); FX_LOGS(INFO) << "Wrote value at key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply(fit::ok()); } void Export(ExportRequest& request, ExportCompleter::Sync& completer) override { FX_LOGS(INFO) << "Export request received"; completer.Reply(Export(std::move(request.empty()))); FX_LOGS(INFO) << "Export response sent"; } void handle_unknown_method( fidl::UnknownMethodMetadata<examples_keyvaluestore_supportexports::Store> metadata, fidl::UnknownMethodCompleter::Sync& completer) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << metadata.method_ordinal; } private: using ExportError = ::examples_keyvaluestore_supportexports::ExportError; using Exportable = ::examples_keyvaluestore_supportexports::Exportable; using Item = ::examples_keyvaluestore_supportexports::Item; fit::result<ExportError, zx::vmo> Export(zx::vmo vmo) { if (key_value_store_.empty()) { return fit::error(ExportError::kEmpty); } Exportable exportable; std::vector<Item> items; items.reserve(key_value_store_.size()); for (const auto& [k, v] : key_value_store_) { items.push_back(Item{{.key = k, .value = v}}); } std::sort(items.begin(), items.end(), [](const Item& a, const Item& b) { return a.key() < b.key(); }); exportable.items(std::move(items)); fit::result encoded = fidl::Persist(exportable); if (encoded.is_error()) { FX_LOGS(ERROR) << "Failed to encode in persistence convention: " << encoded.error_value(); return fit::error(ExportError::kUnknown); } size_t content_size = 0; if (vmo.get_prop_content_size(&content_size) != ZX_OK) { return fit::error(ExportError::kUnknown); } if (encoded->size() > content_size) { return fit::error(ExportError::kStorageTooSmall); } if (vmo.set_prop_content_size(encoded->size()) != ZX_OK) { return fit::error(ExportError::kUnknown); } if (vmo.write(encoded->data(), 0, encoded->size()) != ZX_OK) { return fit::error(ExportError::kUnknown); } return fit::ok(std::move(vmo)); } fidl::ServerBindingRef<examples_keyvaluestore_supportexports::Store> binding_; // The map that serves as the per-connection instance of the key-value store. std::unordered_map<std::string, std::vector<uint8_t>> key_value_store_ = {}; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. This // directory is where the outgoing FIDL protocols are installed so that they can be provided to // other components. component::OutgoingDirectory outgoing = component::OutgoingDirectory(dispatcher); // The `ServeFromStartupInfo()` function sets up the outgoing directory with the startup handle. // The startup handle is a handle provided to every component by the system, so that they can // serve capabilities (e.g. FIDL protocols) to other components. zx::result result = outgoing.ServeFromStartupInfo(); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to serve outgoing directory: " << result.status_string(); return -1; } // Register a handler for components trying to connect to |Store|. result = outgoing.AddUnmanagedProtocol<examples_keyvaluestore_supportexports::Store>( [dispatcher](fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) { // Create an instance of our StoreImpl that destroys itself when the connection closes. new StoreImpl(dispatcher, std::move(server_end)); }); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to add Store protocol: " << result.status_string(); return -1; } // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
C++ (Wire)
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.keyvaluestore.supportexports/cpp/wire.h> #include <lib/async-loop/cpp/loop.h> #include <lib/component/incoming/cpp/protocol.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <examples/fidl/new/key_value_store/support_exports/cpp_wire/client/config.h> #include <src/lib/files/file.h> #include <src/lib/fxl/strings/string_printf.h> int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop and dispatcher. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace. This can fail so it's wrapped in a // |zx::result| and it must be checked for errors. zx::result client_end = component::Connect<examples_keyvaluestore_supportexports::Store>(); if (!client_end.is_ok()) { FX_LOGS(ERROR) << "Synchronous error when connecting to the |Store| protocol: " << client_end.status_string(); return -1; } // Create an asynchronous client using the newly-established connection. fidl::WireClient client(std::move(*client_end), dispatcher); FX_LOGS(INFO) << "Outgoing connection enabled"; for (const auto& key : conf.write_items()) { std::string text; if (!files::ReadFileToString(fxl::StringPrintf("/pkg/data/%s.txt", key.c_str()), &text)) { FX_LOGS(ERROR) << "It looks like the correct `resource` dependency has not been packaged"; break; } auto value = std::vector<uint8_t>(text.begin(), text.end()); client ->WriteItem( {fidl::StringView::FromExternal(key), fidl::VectorView<uint8_t>::FromExternal(value)}) .ThenExactlyOnce( [&](fidl::WireUnownedResult<examples_keyvaluestore_supportexports::Store::WriteItem>& result) { if (!result.ok()) { FX_LOGS(ERROR) << "Unexpected framework error"; } else if (result->is_error()) { FX_LOGS(INFO) << "WriteItem Error: " << fidl::ToUnderlying(result->error_value()); } else { FX_LOGS(INFO) << "WriteItem Success"; } // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // If the `max_export_size` is 0, no export is possible, so just ignore this block. This check // isn't strictly necessary, but does avoid extra work down the line. if (conf.max_export_size() > 0) { // Create a 100Kb VMO to store the resulting export. In a real implementation, we would // likely receive the VMO representing the to-be-written file from file system like vfs of // fxfs. zx::vmo vmo; if (zx_status_t status = zx::vmo::create(conf.max_export_size(), 0, &vmo); status != ZX_OK) { FX_PLOGS(ERROR, status) << "Failed to create VMO"; return -1; } client->Export(std::move(vmo)) .ThenExactlyOnce( [&](fidl::WireUnownedResult<examples_keyvaluestore_supportexports::Store::Export>& result) { // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over, when we return from this callback. loop.Quit(); if (!result.ok()) { FX_LOGS(ERROR) << "Unexpected FIDL framework error: " << result.error(); return; } if (!result->is_ok()) { FX_LOGS(INFO) << "Export Error: " << fidl::ToUnderlying(result->error_value()); return; } FX_LOGS(INFO) << "Export Success"; // Read the exported data (encoded in byte form as persistent FIDL) from the // returned VMO. In a real implementation, instead of reading the VMO, we would // merely forward it to some other storage-handling process. Doing this using a VMO, // rather than FIDL IPC, would save us frivolous reads and writes at each hop. size_t content_size = 0; zx::vmo vmo = std::move(result->value()->filled); if (vmo.get_prop_content_size(&content_size) != ZX_OK) { return; } std::vector<uint8_t> encoded_bytes; encoded_bytes.resize(content_size); if (vmo.read(encoded_bytes.data(), 0, content_size) != ZX_OK) { return; } // Decode the persistent FIDL that was just read from the file. fit::result exportable = fidl::InplaceUnpersist<examples_keyvaluestore_supportexports::wire::Exportable>( cpp20::span(encoded_bytes)); if (exportable.is_error()) { FX_LOGS(ERROR) << "Failed to unpersist: " << exportable.error_value(); return; } if (!exportable->has_items()) { FX_LOGS(INFO) << "Expected items to be set"; return; } auto& items = exportable->items(); // Log some information about the exported data. FX_LOGS(INFO) << "Printing " << items.count() << " exported entries, which are:"; for (const auto& item : items) { FX_LOGS(INFO) << " * " << item.key.get(); } }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.keyvaluestore.supportexports/cpp/wire.h> #include <lib/async-loop/cpp/loop.h> #include <lib/async/cpp/task.h> #include <lib/component/outgoing/cpp/outgoing_directory.h> #include <lib/fidl/cpp/wire/channel.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <algorithm> #include <re2/re2.h> // An implementation of the |Store| protocol. class StoreImpl final : public fidl::WireServer<examples_keyvaluestore_supportexports::Store> { public: // Bind this implementation to a channel. StoreImpl(async_dispatcher_t* dispatcher, fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) : binding_(fidl::BindServer( dispatcher, std::move(server_end), this, [this](StoreImpl* impl, fidl::UnbindInfo info, fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) { if (info.reason() != ::fidl::Reason::kPeerClosedWhileReading) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; })) {} void WriteItem(WriteItemRequestView request, WriteItemCompleter::Sync& completer) override { FX_LOGS(INFO) << "WriteItem request received"; std::string key{request->attempt.key.get()}; std::vector<uint8_t> value{request->attempt.value.begin(), request->attempt.value.end()}; // Validate the key. if (!RE2::FullMatch(key, "^[A-Za-z]\\w+[A-Za-z0-9]$")) { FX_LOGS(INFO) << "Write error: INVALID_KEY, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kInvalidKey)); } // Validate the value. if (value.empty()) { FX_LOGS(INFO) << "Write error: INVALID_VALUE, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kInvalidValue)); } if (key_value_store_.find(key) != key_value_store_.end()) { FX_LOGS(INFO) << "Write error: ALREADY_EXISTS, For key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply( fit::error(examples_keyvaluestore_supportexports::WriteError::kAlreadyExists)); } // Ensure that the value does not already exist in the store. key_value_store_.insert({key, value}); FX_LOGS(INFO) << "Wrote value at key: " << key; FX_LOGS(INFO) << "WriteItem response sent"; return completer.Reply(fit::success()); } void Export(ExportRequestView request, ExportCompleter::Sync& completer) override { FX_LOGS(INFO) << "Export request received"; fit::result result = Export(std::move(request->empty)); if (result.is_ok()) { completer.ReplySuccess(std::move(result.value())); } else { completer.ReplyError(result.error_value()); } FX_LOGS(INFO) << "Export response sent"; } using ExportError = ::examples_keyvaluestore_supportexports::wire::ExportError; using Exportable = ::examples_keyvaluestore_supportexports::wire::Exportable; using Item = ::examples_keyvaluestore_supportexports::wire::Item; fit::result<ExportError, zx::vmo> Export(zx::vmo vmo) { if (key_value_store_.empty()) { return fit::error(ExportError::kEmpty); } fidl::Arena arena; fidl::VectorView<Item> items; items.Allocate(arena, key_value_store_.size()); size_t count = 0; for (auto& [k, v] : key_value_store_) { // Create a wire |Item| object that borrows from |k| and |v|. // Since |k| and |v| are references into the long living |key_value_store_|, // while |items| only live within the current function scope, // this operation is safe. items[count] = Item{ .key = fidl::StringView::FromExternal(k), .value = fidl::VectorView<uint8_t>::FromExternal(v), }; count++; } std::sort(items.begin(), items.end(), [](const Item& a, const Item& b) { return a.key.get() < b.key.get(); }); Exportable exportable = Exportable::Builder(arena).items(items).Build(); fit::result encoded = fidl::Persist(exportable); if (encoded.is_error()) { FX_LOGS(ERROR) << "Failed to encode in persistence convention: " << encoded.error_value(); return fit::error(ExportError::kUnknown); } size_t content_size = 0; if (vmo.get_prop_content_size(&content_size) != ZX_OK) { return fit::error(ExportError::kUnknown); } if (encoded->size() > content_size) { return fit::error(ExportError::kStorageTooSmall); } if (vmo.set_prop_content_size(encoded->size()) != ZX_OK) { return fit::error(ExportError::kUnknown); } if (vmo.write(encoded->data(), 0, encoded->size()) != ZX_OK) { return fit::error(ExportError::kUnknown); } return fit::ok(std::move(vmo)); } void handle_unknown_method( fidl::UnknownMethodMetadata<examples_keyvaluestore_supportexports::Store> metadata, fidl::UnknownMethodCompleter::Sync& completer) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << metadata.method_ordinal; } private: fidl::ServerBindingRef<examples_keyvaluestore_supportexports::Store> binding_; // The map that serves as the per-connection instance of the key-value store. // // Out-of-line references in wire types are always mutable. Thus the // |const std::vector<uint8_t>| from the baseline needs to be changed to // non-const as we're making a vector view pointing to it during |Export|, // even though in practice the value is never mutated. std::unordered_map<std::string, std::vector<uint8_t>> key_value_store_ = {}; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. This // directory is where the outgoing FIDL protocols are installed so that they can be provided to // other components. component::OutgoingDirectory outgoing = component::OutgoingDirectory(dispatcher); // The `ServeFromStartupInfo()` function sets up the outgoing directory with the startup handle. // The startup handle is a handle provided to every component by the system, so that they can // serve capabilities (e.g. FIDL protocols) to other components. zx::result result = outgoing.ServeFromStartupInfo(); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to serve outgoing directory: " << result.status_string(); return -1; } // Register a handler for components trying to connect to |Store|. result = outgoing.AddUnmanagedProtocol<examples_keyvaluestore_supportexports::Store>( [dispatcher](fidl::ServerEnd<examples_keyvaluestore_supportexports::Store> server_end) { // Create an instance of our StoreImpl that destroys itself when the connection closes. new StoreImpl(dispatcher, std::move(server_end)); }); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to add Store protocol: " << result.status_string(); return -1; } // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Protocol end
FIDL recipe: Protocol end
A protocol end represents one endpoint of a channel connection over which some
specified FIDL protocol is spoken. The server side of this connection is the
server_end, while the client side is the client_end.
Protocol ends have a required constraint, specifying the FIDL protocol being
spoken over the connection. For instance, client_end:Foo represents the client
endpoint of a Zircon channel over which all messages exchanged will conform to
methods and events defined in that FIDL protocol, while server_end:Foo
represents the opposite endpoint.
A useful operation for key-value stores is in-order iteration: that is, when given a key, to return a (usually paginated) list of elements that appear after it, in order.
Reasoning
In FIDL, this is best done using an iterator, which is generally implemented as a separate protocol over which this iteration can occur. Using a separate protocol, and therefore a separate channel, has a number of benefits, including de-interleaving the iteration pull requests from other operations done over the main protocol.
The client and server side of the channel connection for protocol P can be
represented as FIDL data types, as a client_end:P and server_end:P,
respectively. These types are collectively known as protocol ends, and
represent the other (non-@discoverable) way of connecting a FIDL client to its
corresponding server: over an existing FIDL connection!
Protocol ends are specific instances of a general FIDL concept: the resource type. A resource type is intended to contain FIDL handles, which necessitates extra restrictions on how the type can be used. The type must be always be unique, as the underlying resource is mediated by some other capability manager (usually the Zircon kernel). Duplicating such a resource via a simple in-memory copy, without involving the manager, is impossible. To prevent such duplication, all resource types in FIDL are always move-only.
Finally, the Get() method of the Iterator protocol itself makes use of a
size constraint on the return payload. This limits the amount of data that may
be transmitted in a single pull, allowing for some measure of resource use
control. It also creates a natural pagination boundary: rather than a giant dump
of all of the results at once, the server only needs to prepare small batches at
a time.
Implementation
The FIDL, CML, and realm interface definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.additerator; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value vector<byte>:64000; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// An enumeration of things that may go wrong when trying to create an iterator. type IterateConnectionError = flexible enum { /// The starting key was not found. UNKNOWN_START_AT = 1; }; /// A key-value store which supports insertion and iteration. @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; /// Iterates over the items in the store, using lexicographic ordering over the keys. /// /// The [`iterator`] is [pipelined][pipelining] to the server, such that the client can /// immediately send requests over the new connection. /// /// [pipelining]: https://fuchsia.dev/fuchsia-src/development/api/fidl?hl=en#request-pipelining flexible Iterate(resource struct { /// If present, requests to start the iteration at this item. starting_at string:<128, optional>; /// The [`Iterator`] server endpoint. The client creates both ends of the channel and /// retains the `client_end` locally to use for pulling iteration pages, while sending the /// `server_end` off to be fulfilled by the server. iterator server_end:Iterator; }) -> () error IterateConnectionError; }; /// An iterator for the key-value store. Note that this protocol makes no guarantee of atomicity - /// the values may change between pulls from the iterator. Unlike the `Store` protocol above, this /// protocol is not `@discoverable`: it is not independently published by the component that /// implements it, but rather must have one of its two protocol ends transmitted over an existing /// FIDL connection. /// /// As is often the case with iterators, the client indicates that they are done with an instance of /// the iterator by simply closing their end of the connection. /// /// Since the iterator is associated only with the Iterate method, it is declared as closed rather /// than open. This is because changes to how iteration works are more likely to require replacing /// the Iterate method completely (which is fine because that method is flexible) rather than /// evolving the Iterator protocol. closed protocol Iterator { /// Gets the next batch of keys. /// /// The client pulls keys rather than having the server proactively push them, to implement /// [flow control][flow-control] over the messages. /// /// [flow-control]: /// https://fuchsia.dev/fuchsia-src/development/api/fidl?hl=en#prefer_pull_to_push strict Get() -> (struct { /// A list of keys. If the iterator has reached the end of iteration, the list will be /// empty. The client is expected to then close the connection. entries vector<string:128>:10; }); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.additerator.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A key to iterate from, after all items in `write_items` have been written. iterate_from: { type: "string", max_size: 64, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.additerator.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.additerator.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.additerator.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use config::Config; use fuchsia_component::client::connect_to_protocol; use std::{thread, time}; use fidl::endpoints::create_proxy; use fidl_examples_keyvaluestore_additerator::{Item, IteratorMarker, StoreMarker}; use futures::join; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; match store.write_item(&Item { key: key, value: value.into_bytes() }).await? { Ok(_) => println!("WriteItem Success"), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } if !config.iterate_from.is_empty() { // This helper creates a channel, and returns two protocol ends: the `client_end` is already // conveniently bound to the correct FIDL protocol, `Iterator`, while the `server_end` is // unbound and ready to be sent over the wire. let (iterator, server_end) = create_proxy::<IteratorMarker>(); // There is no need to wait for the iterator to connect before sending the first `Get()` // request - since we already hold the `client_end` of the connection, we can start queuing // requests on it immediately. let connect_to_iterator = store.iterate(Some(config.iterate_from.as_str()), server_end); let first_get = iterator.get(); // Wait until both the connection and the first request resolve - an error in either case // triggers an immediate resolution of the combined future. let (connection, first_page) = join!(connect_to_iterator, first_get); // Handle any connection error. If this has occurred, it is impossible for the first `Get()` // call to have resolved successfully, so check this error first. if let Err(err) = connection.context("Could not connect to Iterator")? { println!("Iterator Connection Error: {}", err.into_primitive()); } else { println!("Iterator Connection Success"); // Consecutively repeat the `Get()` request if the previous response was not empty. let mut entries = first_page.context("Could not get page from Iterator")?; while !&entries.is_empty() { for entry in entries.iter() { println!("Iterator Entry: {}", entry); } entries = iterator.get().await.context("Could not get page from Iterator")?; } } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fuchsia_component::server::ServiceFs; use futures::prelude::*; use lazy_static::lazy_static; use regex::Regex; use fidl_examples_keyvaluestore_additerator::{ Item, IterateConnectionError, IteratorRequest, IteratorRequestStream, StoreRequest, StoreRequestStream, WriteError, }; use fuchsia_async as fasync; use std::collections::btree_map::Entry; use std::collections::BTreeMap; use std::ops::Bound::*; use std::sync::{Arc, Mutex}; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } /// Handler for the `WriteItem` method. fn write_item(store: &mut BTreeMap<String, Vec<u8>>, attempt: Item) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Validate the value. if attempt.value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", attempt.key); return Err(WriteError::InvalidValue); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote value at key: {}", entry.key()); entry.insert(attempt.value); Ok(()) } } } /// Handler for the `Iterate` method, which deals with validating that the requested start position /// exists, and then sets up the asynchronous side channel for the actual iteration to occur over. fn iterate( store: Arc<Mutex<BTreeMap<String, Vec<u8>>>>, starting_at: Option<String>, stream: IteratorRequestStream, ) -> Result<(), IterateConnectionError> { // Validate that the starting key, if supplied, actually exists. if let Some(start_key) = starting_at.clone() { if !store.lock().unwrap().contains_key(&start_key) { return Err(IterateConnectionError::UnknownStartAt); } } // Spawn a detached task. This allows the method call to return while the iteration continues in // a separate, unawaited task. fasync::Task::spawn(async move { // Serve the iteration requests. Note that access to the underlying store is behind a // contended `Mutex`, meaning that the iteration is not atomic: page contents could shift, // change, or disappear entirely between `Get()` requests. stream .map(|result| result.context("failed request")) .try_fold( match starting_at { Some(start_key) => Included(start_key), None => Unbounded, }, |mut lower_bound, request| async { match request { IteratorRequest::Get { responder } => { println!("Iterator page request received"); // The `page_size` should be kept in sync with the size constraint on // the iterator's response, as defined in the FIDL protocol. static PAGE_SIZE: usize = 10; // An iterator, beginning at `lower_bound` and tracking the pagination's // progress through iteration as each page is pulled by a client-sent // `Get()` request. let held_store = store.lock().unwrap(); let mut entries = held_store.range((lower_bound.clone(), Unbounded)); let mut current_page = vec![]; for _ in 0..PAGE_SIZE { match entries.next() { Some(entry) => { current_page.push(entry.0.clone()); } None => break, } } // Update the `lower_bound` - either inclusive of the next item in the // iteration, or exclusive of the last seen item if the iteration has // finished. This `lower_bound` will be passed to the next request // handler as its starting point. lower_bound = match entries.next() { Some(next) => Included(next.0.clone()), None => match current_page.last() { Some(tail) => Excluded(tail.clone()), None => lower_bound, }, }; // Send the page. At the end of this scope, the `held_store` lock gets // dropped, and therefore released. responder.send(¤t_page).context("error sending reply")?; println!("Iterator page sent"); } } Ok(lower_bound) }, ) .await .ok(); }) .detach(); Ok(()) } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. // // Note that we now use an `Arc<Mutex<BTreeMap>>`, replacing the previous `RefCell<HashMap>`. // The `BTreeMap` is used because we want an ordered map, to better facilitate iteration. The // `Arc<Mutex<...>>` is used because there are now multiple async tasks accessing the: one main // task which handles communication over the protocol, and one additional task per iterator // protocol. `Arc<Mutex<...>>` is the simplest way to synchronize concurrent access between // these racing tasks. let store = &Arc::new(Mutex::new(BTreeMap::<String, Vec<u8>>::new())); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.clone().lock().unwrap(), attempt)) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::Iterate { starting_at, iterator, responder } => { println!("Iterate request received"); // The `iterate` handler does a quick check to see that the request is valid, // then spins up a separate worker task to serve the newly minted `Iterator` // protocol instance, allowing this call to return immediately and continue the // request stream with other work. responder .send(iterate(store.clone(), starting_at, iterator.into_stream())) .context("error sending reply")?; println!("Iterate response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Protocol
FIDL recipe: Protocol
A protocol describes a set of methods that can be invoked by sending messages over a channel. It describes the interface over which FIDL data may be exchanged between a client and a server.
In this example, you will create a basic calculator server & client which shows the fundamental setup needed to first define and then serve and consume a FIDL protocol.
First, you will define the interface definitions and test harness. The
interface definition (the .fidl file itself) is the starting point for any new
FIDL protocol. Additionally, the calculator includes the necessary CML and realm
definitions to create a client-server pattern which can be used as project
scaffolding for arbitrary implementations.
See below for the FIDL code:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. // The namespace for this FIDL protocol. This namespace is how both consumers (clients) and providers (servers) reference this protocol. library examples.calculator.baseline; // @discoverable indicates 'Calculator' is a protocol that will be served under the examples.calculator.baseline libarary namespace. https://fuchsia.dev/fuchsia-src/reference/fidl/language/attributes#discoverable . If @discoverable is missing, it will lead to a compile time error when trying to import the library. @discoverable // A limited-functionality calculator 'protocol' that adds and subtracts integers. open protocol Calculator { // Takes as input a struct with two integers, and returns their sum: (a+b)=sum. This method is infallible (no errors can be generated) as two int32's cannot overflow a result type of int64. flexible Add(struct { a int32; b int32; }) -> (struct { sum int64; }); // Takes as input a struct with two integers, and returns their difference: (a-b)=difference. This method is infallible (no errors can be generated) as two int32's cannot overflow a result type of int64. flexible Subtract(struct { a int32; b int32; }) -> (struct { difference int64; }); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.calculator.baseline.Calculator" }, ], config: {}, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.calculator.baseline.Calculator" }, ], expose: [ { protocol: "examples.calculator.baseline.Calculator", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.calculator.baseline.Calculator", from: "#server", to: "#client", }, // Route logging support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// TODO(https://fxbug.dev/42063075): Rust implementation.Server
// TODO(https://fxbug.dev/42063075): Rust implementation.C++ (Natural)
Client
// TODO(https://fxbug.dev/42063075): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42063075): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42063075): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42063075): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42063075): HLCPP implementation.Server
// TODO(https://fxbug.dev/42063075): HLCPP implementation.Creating a FIDL protocol from the ground up as is shown in this example can be a more common scenario for certain developers, such as platform developers. However, other types of developers also benefit from learning how to construct a FIDL protocol even if they won't typically do so. This helps you learn how everything about FIDL fits together, including the syntax, grammar, language features, how to serve and consume a given FIDL protocol, and how the build system works. For next steps, the examples which follow this baseline show how to extend an existing FIDL protocol, which is expected to be a fairly common practice.
Recursive type
FIDL recipe: Recursive type
Recursive types are types that transitively refer to themselves in their own definition. This can happen when a type makes a direct reference to itself, or when it refers to some chain of types that transitively include it. For instance, in the classic tree data structure, each node may either contain only data (a "leaf"), or data and references to more child nodes (a "branch"). In the latter case, the node recursively contains a nested tree definition, repeating to as great of a depth as necessary.
FIDL supports recursive types, as long as at least one chain in the includes cycle (in other words, the chain of type definitions that leads back to the original type) is optional. If no type in the chain were optional, the type would be unencodable, as each instance of the type would require at least one more inside of it, ad infinitum.
In this variant, we allow our key-value store to take other key-value stores as
members. In short, we turn it into a tree. We do this by replacing the original
definition of value with one that utilizes a two-member union: one variant
stores leaf nodes using the same vector<byte> type as before, while the other
stores branch nodes in the form of other nested stores.
Reasoning
Here, we see several uses of optionality, whereby we can declare a type that may or may not exist. There are three flavors of optionality in FIDL:
- Types that have are always stored
out-of-line
on the wire, and thus have a builtin way to describe "absentness" via the
null envelope. Enabling
optionality for these types doesn't affect the wire shape of messages they are
included in - it simply changes which values are valid for that particular
type. The
union,vector<T>,client_end,server_end, andzx.Handletypes can all be made optional via the addition of the:optionalconstraint. By making ourvalueunionoptional, we are able to introduce a canonical "null" entry, in the form of an absentvalue. This means that emptybytesand absent/emptystoreproperties are invalid values. - Unlike the aforementioned types, the
structlayout has no extra space where a null header can be stored. Because of this, it needs to be wrapped in an envelope, changing the on-the-wire shape of the message it is being included in. To ensure that this wire-modifying effect easily legible, theItemstructtype must be wrapped in abox<T>type template. - Finally,
tablelayouts are always optional. An absenttableis simply one with none of its members set.
Trees are a naturally self-referential data structure: any node in the tree may
contain a leaf with pure data (in this case, a string), or a sub-tree with more
nodes. This requires recursion: the definition of Item is now transitively
dependent on itself! Representing recursive types in FIDL can be a bit tricky,
especially because support is currently somewhat
limited. We can support such types as long as there is
at least one optional type in the cycle created by the self-reference. For
instance, here we define the items struct member to be a box<Item>,
thereby breaking the includes cycle.
These changes also make heavy use of anonymous types, or types whose
declarations are inlined at their sole point of use, rather than being named,
top-level type declarations of their own. By default, the names of anonymous
types in the generated language bindings are taken from their local context. For
instance, the newly introduced flexible union takes on its owning member's
name Value, the newly introduced struct would become Store, and so on.
Because this heuristic can sometimes cause collisions, FIDL provides an escape
hatch by allowing the author to manually override an anonymous type's generated
name. This is done via the @generated_name attribute, which allows one to
change the name generated by backends. We can use one here, where the would-be
Store type is renamed to NestedStore to prevent a name collision with the
protocol declaration that uses that same name.
Implementation
The FIDL, CML, and realm interface definitions are modified as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.supporttrees; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value strict union { // Keep the original `bytes` as one of the options in the new union. All leaf nodes in the // tree must be `bytes`, or absent unions (representing empty). Empty byte arrays are // disallowed. 1: bytes vector<byte>:64000; // Allows a store within a store, thereby turning our flat key-value store into a tree // thereof. Note the use of `@generated_name` to prevent a type-name collision with the // `Store` protocol below, and the use of `box<T>` to ensure that there is a break in the // chain of recursion, thereby allowing `Item` to include itself in its own definition. // // This is a table so that added fields, like for example a `hash`, can be easily added in // the future. 2: store @generated_name("nested_store") table { 1: items vector<box<Item>>; }; }:optional; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// A very basic key-value store. @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.supporttrees.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A newline separated list nested entries. The first line should be the key // for the nested store, and each subsequent entry should be a pointer to a text file // containing the string value. The name of that text file (without the `.txt` suffix) will // serve as the entries key. write_nested: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A list of keys, all of which will be populated as null entries. write_null: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.supporttrees.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.supporttrees.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.supporttrees.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, config::Config, fidl_examples_keyvaluestore_supporttrees::{Item, NestedStore, StoreMarker, Value}, fuchsia_component::client::connect_to_protocol, std::{thread, time}, }; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; let res = store .write_item(&Item { key: key.clone(), value: Some(Box::new(Value::Bytes(value.into_bytes()))), }) .await; match res? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // Add nested entries to the key-value store as well. The entries are strings, where the first // line is the key of the entry, and each subsequent entry should be a pointer to a text file // containing the string value. The name of that text file (without the `.txt` suffix) will // serve as the entries key. for spec in config.write_nested.into_iter() { let mut items = vec![]; let mut nested_store = NestedStore::default(); let mut lines = spec.split("\n"); let key = lines.next().unwrap(); // For each entry, make a new entry in the `NestedStore` being built. for entry in lines { let path = format!("/pkg/data/{}.txt", entry); let contents = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; items.push(Some(Box::new(Item { key: entry.to_string(), value: Some(Box::new(Value::Bytes(contents.into()))), }))); } nested_store.items = Some(items); // Send the `NestedStore`, represented as a vector of values. let res = store .write_item(&Item { key: key.to_string(), value: Some(Box::new(Value::Store(nested_store))), }) .await; match res? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // Each entry in this list is a null value in the store. for key in config.write_null.into_iter() { match store.write_item(&Item { key: key.to_string(), value: None }).await? { Ok(_) => println!("WriteItem Success at key: {}", key), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. // Note: For the clarity of this example, allow code to be unused. #![allow(dead_code)] use { anyhow::{Context as _, Error}, fidl_examples_keyvaluestore_supporttrees::{ Item, StoreRequest, StoreRequestStream, Value, WriteError, }, fuchsia_component::server::ServiceFs, futures::prelude::*, lazy_static::lazy_static, regex::Regex, std::cell::RefCell, std::collections::hash_map::Entry, std::collections::HashMap, std::str::from_utf8, }; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } // A representation of a key-value store that can contain an arbitrarily deep nesting of other // key-value stores. #[allow(clippy::box_collection, reason = "mass allow for https://fxbug.dev/381896734")] enum StoreNode { Leaf(Option<Vec<u8>>), Branch(Box<HashMap<String, StoreNode>>), } /// Recursive item writer, which takes a `StoreNode` that may not necessarily be the root node, and /// writes an entry to it. fn write_item( store: &mut HashMap<String, StoreNode>, attempt: Item, path: &str, ) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { let key = format!("{}{}", &path, entry.key()); match attempt.value { // Null entries are allowed. None => { println!("Wrote value: NONE at key: {}", key); entry.insert(StoreNode::Leaf(None)); } Some(value) => match *value { // If this is a nested store, recursively make a new store to insert at this // position. Value::Store(entry_list) => { // Validate the value - absent stores, items lists with no children, or any // of the elements within that list being empty boxes, are all not allowed. if entry_list.items.is_some() { let items = entry_list.items.unwrap(); if !items.is_empty() && items.iter().all(|i| i.is_some()) { let nested_path = format!("{}/", key); let mut nested_store = HashMap::<String, StoreNode>::new(); for item in items.into_iter() { write_item(&mut nested_store, *item.unwrap(), &nested_path)?; } println!("Created branch at key: {}", key); entry.insert(StoreNode::Branch(Box::new(nested_store))); return Ok(()); } } println!("Write error: INVALID_VALUE, For key: {}", key); return Err(WriteError::InvalidValue); } // This is a simple leaf node on this branch. Value::Bytes(value) => { // Validate the value. if value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", key); return Err(WriteError::InvalidValue); } println!("Wrote key: {}, value: {:?}", key, from_utf8(&value).unwrap()); entry.insert(StoreNode::Leaf(Some(value))); } }, } Ok(()) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, StoreNode>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.borrow_mut(), attempt, "")) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Resource type
FIDL recipe: Resource type
A FIDL resource type is a type that is intended to transitively carry handles. Because FIDL handles are unique references to capabilities, any type that contains one inherits this behavior: it too cannot be copied. In this way, resourceness is infectious: if a value type becomes a resource type, all types that transitively include it must do so as well.
A useful operation for key-value stores is in-order iteration: that is, when given a key, to return a (usually paginated) list of elements that appear after it, in order.
Reasoning
In FIDL, this is best done using an iterator, which is generally implemented as a separate protocol over which this iteration can occur. Using a separate protocol, and therefore a separate channel, has a number of benefits, including de-interleaving the iteration pull requests from other operations done over the main protocol.
The client and server side of the channel connection for protocol P can be
represented as FIDL data types, as a client_end:P and server_end:P,
respectively. These types are collectively known as protocol ends, and
represent the other (non-@discoverable) way of connecting a FIDL client to its
corresponding server: over an existing FIDL connection!
Protocol ends are specific instances of a general FIDL concept: the resource type. A resource type is intended to contain FIDL handles, which necessitates extra restrictions on how the type can be used. The type must be always be unique, as the underlying resource is mediated by some other capability manager (usually the Zircon kernel). Duplicating such a resource via a simple in-memory copy, without involving the manager, is impossible. To prevent such duplication, all resource types in FIDL are always move-only.
Finally, the Get() method of the Iterator protocol itself makes use of a
size constraint on the return payload. This limits the amount of data that may
be transmitted in a single pull, allowing for some measure of resource use
control. It also creates a natural pagination boundary: rather than a giant dump
of all of the results at once, the server only needs to prepare small batches at
a time.
Implementation
The FIDL, CML, and realm interface definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.additerator; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value vector<byte>:64000; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// An enumeration of things that may go wrong when trying to create an iterator. type IterateConnectionError = flexible enum { /// The starting key was not found. UNKNOWN_START_AT = 1; }; /// A key-value store which supports insertion and iteration. @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; /// Iterates over the items in the store, using lexicographic ordering over the keys. /// /// The [`iterator`] is [pipelined][pipelining] to the server, such that the client can /// immediately send requests over the new connection. /// /// [pipelining]: https://fuchsia.dev/fuchsia-src/development/api/fidl?hl=en#request-pipelining flexible Iterate(resource struct { /// If present, requests to start the iteration at this item. starting_at string:<128, optional>; /// The [`Iterator`] server endpoint. The client creates both ends of the channel and /// retains the `client_end` locally to use for pulling iteration pages, while sending the /// `server_end` off to be fulfilled by the server. iterator server_end:Iterator; }) -> () error IterateConnectionError; }; /// An iterator for the key-value store. Note that this protocol makes no guarantee of atomicity - /// the values may change between pulls from the iterator. Unlike the `Store` protocol above, this /// protocol is not `@discoverable`: it is not independently published by the component that /// implements it, but rather must have one of its two protocol ends transmitted over an existing /// FIDL connection. /// /// As is often the case with iterators, the client indicates that they are done with an instance of /// the iterator by simply closing their end of the connection. /// /// Since the iterator is associated only with the Iterate method, it is declared as closed rather /// than open. This is because changes to how iteration works are more likely to require replacing /// the Iterate method completely (which is fine because that method is flexible) rather than /// evolving the Iterator protocol. closed protocol Iterator { /// Gets the next batch of keys. /// /// The client pulls keys rather than having the server proactively push them, to implement /// [flow control][flow-control] over the messages. /// /// [flow-control]: /// https://fuchsia.dev/fuchsia-src/development/api/fidl?hl=en#prefer_pull_to_push strict Get() -> (struct { /// A list of keys. If the iterator has reached the end of iteration, the list will be /// empty. The client is expected to then close the connection. entries vector<string:128>:10; }); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.additerator.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A key to iterate from, after all items in `write_items` have been written. iterate_from: { type: "string", max_size: 64, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.additerator.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.additerator.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.additerator.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use config::Config; use fuchsia_component::client::connect_to_protocol; use std::{thread, time}; use fidl::endpoints::create_proxy; use fidl_examples_keyvaluestore_additerator::{Item, IteratorMarker, StoreMarker}; use futures::join; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; match store.write_item(&Item { key: key, value: value.into_bytes() }).await? { Ok(_) => println!("WriteItem Success"), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } if !config.iterate_from.is_empty() { // This helper creates a channel, and returns two protocol ends: the `client_end` is already // conveniently bound to the correct FIDL protocol, `Iterator`, while the `server_end` is // unbound and ready to be sent over the wire. let (iterator, server_end) = create_proxy::<IteratorMarker>(); // There is no need to wait for the iterator to connect before sending the first `Get()` // request - since we already hold the `client_end` of the connection, we can start queuing // requests on it immediately. let connect_to_iterator = store.iterate(Some(config.iterate_from.as_str()), server_end); let first_get = iterator.get(); // Wait until both the connection and the first request resolve - an error in either case // triggers an immediate resolution of the combined future. let (connection, first_page) = join!(connect_to_iterator, first_get); // Handle any connection error. If this has occurred, it is impossible for the first `Get()` // call to have resolved successfully, so check this error first. if let Err(err) = connection.context("Could not connect to Iterator")? { println!("Iterator Connection Error: {}", err.into_primitive()); } else { println!("Iterator Connection Success"); // Consecutively repeat the `Get()` request if the previous response was not empty. let mut entries = first_page.context("Could not get page from Iterator")?; while !&entries.is_empty() { for entry in entries.iter() { println!("Iterator Entry: {}", entry); } entries = iterator.get().await.context("Could not get page from Iterator")?; } } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fuchsia_component::server::ServiceFs; use futures::prelude::*; use lazy_static::lazy_static; use regex::Regex; use fidl_examples_keyvaluestore_additerator::{ Item, IterateConnectionError, IteratorRequest, IteratorRequestStream, StoreRequest, StoreRequestStream, WriteError, }; use fuchsia_async as fasync; use std::collections::btree_map::Entry; use std::collections::BTreeMap; use std::ops::Bound::*; use std::sync::{Arc, Mutex}; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } /// Handler for the `WriteItem` method. fn write_item(store: &mut BTreeMap<String, Vec<u8>>, attempt: Item) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Validate the value. if attempt.value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", attempt.key); return Err(WriteError::InvalidValue); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote value at key: {}", entry.key()); entry.insert(attempt.value); Ok(()) } } } /// Handler for the `Iterate` method, which deals with validating that the requested start position /// exists, and then sets up the asynchronous side channel for the actual iteration to occur over. fn iterate( store: Arc<Mutex<BTreeMap<String, Vec<u8>>>>, starting_at: Option<String>, stream: IteratorRequestStream, ) -> Result<(), IterateConnectionError> { // Validate that the starting key, if supplied, actually exists. if let Some(start_key) = starting_at.clone() { if !store.lock().unwrap().contains_key(&start_key) { return Err(IterateConnectionError::UnknownStartAt); } } // Spawn a detached task. This allows the method call to return while the iteration continues in // a separate, unawaited task. fasync::Task::spawn(async move { // Serve the iteration requests. Note that access to the underlying store is behind a // contended `Mutex`, meaning that the iteration is not atomic: page contents could shift, // change, or disappear entirely between `Get()` requests. stream .map(|result| result.context("failed request")) .try_fold( match starting_at { Some(start_key) => Included(start_key), None => Unbounded, }, |mut lower_bound, request| async { match request { IteratorRequest::Get { responder } => { println!("Iterator page request received"); // The `page_size` should be kept in sync with the size constraint on // the iterator's response, as defined in the FIDL protocol. static PAGE_SIZE: usize = 10; // An iterator, beginning at `lower_bound` and tracking the pagination's // progress through iteration as each page is pulled by a client-sent // `Get()` request. let held_store = store.lock().unwrap(); let mut entries = held_store.range((lower_bound.clone(), Unbounded)); let mut current_page = vec![]; for _ in 0..PAGE_SIZE { match entries.next() { Some(entry) => { current_page.push(entry.0.clone()); } None => break, } } // Update the `lower_bound` - either inclusive of the next item in the // iteration, or exclusive of the last seen item if the iteration has // finished. This `lower_bound` will be passed to the next request // handler as its starting point. lower_bound = match entries.next() { Some(next) => Included(next.0.clone()), None => match current_page.last() { Some(tail) => Excluded(tail.clone()), None => lower_bound, }, }; // Send the page. At the end of this scope, the `held_store` lock gets // dropped, and therefore released. responder.send(¤t_page).context("error sending reply")?; println!("Iterator page sent"); } } Ok(lower_bound) }, ) .await .ok(); }) .detach(); Ok(()) } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. // // Note that we now use an `Arc<Mutex<BTreeMap>>`, replacing the previous `RefCell<HashMap>`. // The `BTreeMap` is used because we want an ordered map, to better facilitate iteration. The // `Arc<Mutex<...>>` is used because there are now multiple async tasks accessing the: one main // task which handles communication over the protocol, and one additional task per iterator // protocol. `Arc<Mutex<...>>` is the simplest way to synchronize concurrent access between // these racing tasks. let store = &Arc::new(Mutex::new(BTreeMap::<String, Vec<u8>>::new())); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.clone().lock().unwrap(), attempt)) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::Iterate { starting_at, iterator, responder } => { println!("Iterate request received"); // The `iterate` handler does a quick check to see that the request is valid, // then spins up a separate worker task to serve the newly minted `Iterator` // protocol instance, allowing this call to return immediately and continue the // request stream with other work. responder .send(iterate(store.clone(), starting_at, iterator.into_stream())) .context("error sending reply")?; println!("Iterate response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Scalar type
FIDL recipe: Scalar type
FIDL's scalar types is a category of types that contains all of its builtin
primitives, plus the
builtin non-primitive string type.
Reasoning
The key-value store baseline example's implementation was a good starting point, but one major drawback is that data is stored as raw bytes. FIDL is a richly typed language. Forcing data that is for instance a UTF-8 string to be stored as an untyped byte array erases this valuable type information for readers of the *.fidl file, as well as for programmers using bindings generated from it.
Implementation
The main goal of this change is to replace the baseline case's vector<byte>
typed value member with a union that stores many possible types. In fact, as
of this change a good survey of FIDL's
value types is on
offer:
- All of FIDL's builtin scalar types are used as variants in the
Valueunion:bool,uint8,uint16,uint32,uint64,int8,int16,int32,int64,float32, andfloat64(also known as FIDL's primitive types), as well asstring. - This
unionalso features uses of FIDL's builtinarray<T, N>andvector<T>type templates. - All of FIDL's type layouts, namely
bits,enum,table,union, andstruct, are utilized in this example at least once.
The request and response payloads used for WriteItem have also been changed
from structs to a named table and an inlined flexible union, respectively.
In fact, any of these three layouts may be used a request/response payload. The
latter two, known as table payloads and *union payloads, respectively, are
preferred in all but the most message size sensitive cases. This is because they
are much easier to extend in the future in a binary compatible way.
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.usegenericvalues; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value Value; }; // Because the `Value` must be used both in the request and the response, we give it its own named // type. The type is a `union` of all possible data types that we take as values, and is marked // `flexible` to allow for the easy addition of new data types in the future. type Value = flexible union { // Keep the original `bytes` as one of the options in the new union. 1: bytes vector<byte>:64000; // A `string` is very similar to `vector<byte>` on the wire, with the extra constraint that // it enforces that it enforces that the byte vector in question is valid UTF-8. 2: string string:64000; // All of FIDL's primitive types. 3: bool bool; 4: uint8 uint8; 5: int8 int8; 6: uint16 uint16; 7: int16 int16; 8: uint32 uint32; 9: int32 int32; 10: float32 float32; 11: uint64 uint64; 12: int64 int64; 13: float64 float64; // FIDL does not natively support 128-bit integer types, so we have to define our own // representations. 14: uint128 array<uint64, 2>; }; // Because we now supoprt a richer range of types as values in our store, it is helpful to use a // `flexible`, and therefore evolvable, `bits` type to store write options. type WriteOptions = flexible bits : uint8 { // This flag allows us to overwrite existing data when there is a collision, rather than failing // with an `WriteError.ALREADY_EXISTS`. OVERWRITE = 0b1; // This flag allows us to concatenate to existing data when there is a collision, rather than // failing with an `WriteError.ALREADY_EXISTS`. "Concatenation" means addition for the numeric // variants and appending to the `bytes`/`string` variants. If no existing data can be found, we // "concatenate" to default values of zero and an empty vector, respectively. Attempting to // concatenate to an existing variant of a different type will return a // `WriteError.INVALID_VALUE` error. CONCAT = 0b10; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// A very basic key-value store. @discoverable open protocol Store { /// Writes an item to the store. /// /// Since the value stored in the key-value store can now be different from the input (if the /// `WriteOptions.CONCAT` flag is set), we need to return the resulting `Value` to the /// requester. /// /// We use an (anonymous) `table` and a (named) `flexible union` as the request and response /// payload, respectively, to allow for easier future evolution. Both of these types are /// `flexible`, meaning that adding or removing members is binary-compatible. This makes them /// much easier to evolve that the `struct` types that were previously used, which cannot be /// changed after release without breaking ABI. flexible WriteItem(table { 1: attempt Item; 2: options WriteOptions; }) -> (Value) error WriteError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store" }, ], config: { // A vector of values for every easily representible type in our key-value store. For // brevity's sake, the 8, 16, and 32 bit integer types and booleans are omitted. // // TODO(https://fxbug.dev/42178362): It would absolve individual language implementations of a great // deal of string parsing if we were able to use all FIDL constructs directly here. In // particular, floats and nested types are very difficult to represent, and have been // excluded from this example for the time being. set_concat_option: { type: "bool" }, set_overwrite_option: { type: "bool" }, write_bytes: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, write_strings: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, write_uint64s: { type: "vector", max_count: 16, element: { type: "uint64" }, }, write_int64s: { type: "vector", max_count: 16, element: { type: "int64" }, }, // Note: due to the limitation of structured config not allowing vectors nested in vectors, // we only set the lower half of the uint128 for simplicity's sake. write_uint128s: { type: "vector", max_count: 16, element: { type: "uint64" }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.usegenericvalues.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, config::Config, fidl_examples_keyvaluestore_usegenericvalues::{ Item, StoreMarker, StoreProxy, StoreWriteItemRequest, Value, WriteOptions, }, fuchsia_component::client::connect_to_protocol, std::{thread, time}, }; // A helper function to sequentially write a single item to the key-value store and print a log when // successful. async fn write_next_item( store: &StoreProxy, key: &str, value: Value, options: WriteOptions, ) -> Result<(), Error> { // Create an empty request payload using `::default()`. let mut req = StoreWriteItemRequest::default(); req.options = Some(options); // Fill in the `Item` we will be attempting to write. println!("WriteItem request sent: key: {}, value: {:?}", &key, &value); req.attempt = Some(Item { key: key.to_string(), value: value }); // Send and async `WriteItem` request to the server. match store.write_item(&req).await.context("Error sending request")? { Ok(value) => println!("WriteItem response received: {:?}", &value), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } Ok(()) } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // All of our requests will have the same bitflags set. Pull these settings from the config. let mut options = WriteOptions::empty(); options.set(WriteOptions::OVERWRITE, config.set_overwrite_option); options.set(WriteOptions::CONCAT, config.set_concat_option); // The structured config provides one input for most data types that can be stored in the data // store. Iterate through those inputs in the order we see them in the FIDL file. // // Note that FIDL unions are rendered as enums in Rust; for example, the `Value` union has now // become a `Value` Rust enum, with each member taking exactly one argument. for value in config.write_bytes.into_iter() { write_next_item(&store, "bytes", Value::Bytes(value.into()), options).await?; } for value in config.write_strings.into_iter() { write_next_item(&store, "string", Value::String(value), options).await?; } for value in config.write_uint64s.into_iter() { write_next_item(&store, "uint64", Value::Uint64(value), options).await?; } for value in config.write_int64s.into_iter() { write_next_item(&store, "int64", Value::Int64(value), options).await?; } for value in config.write_uint128s.into_iter() { write_next_item(&store, "uint128", Value::Uint128([0, value]), options).await?; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fuchsia_component::server::ServiceFs; use futures::prelude::*; use lazy_static::lazy_static; use regex::Regex; use std::cell::RefCell; use std::collections::hash_map::Entry; use std::collections::HashMap; use fidl_examples_keyvaluestore_usegenericvalues::{ Item, StoreRequest, StoreRequestStream, Value, WriteError, WriteOptions, }; use std::collections::hash_map::OccupiedEntry; use std::ops::Add; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } /// Sums any numeric type. fn sum<T: Add + Add<Output = T> + Copy>(operands: [T; 2]) -> T { operands[0] + operands[1] } /// Clones and inserts an entry, so that the original (now concatenated) copy may be returned in the /// response. fn write(inserting: Value, mut entry: OccupiedEntry<'_, String, Value>) -> Value { entry.insert(inserting.clone()); println!("Wrote key: {}, value: {:?}", entry.key(), &inserting); inserting } /// Handler for the `WriteItem` method. fn write_item( store: &mut HashMap<String, Value>, attempt: Item, options: &WriteOptions, ) -> Result<Value, WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY for key: {}", attempt.key); return Err(WriteError::InvalidKey); } match store.entry(attempt.key) { Entry::Occupied(entry) => { // The `CONCAT` flag supersedes the `OVERWRITE` flag, so check it first. if options.contains(WriteOptions::CONCAT) { match entry.get() { Value::Bytes(old) => { if let Value::Bytes(new) = attempt.value { let mut combined = old.clone(); combined.extend(new); return Ok(write(Value::Bytes(combined), entry)); } } Value::String(old) => { if let Value::String(new) = attempt.value { return Ok(write(Value::String(format!("{}{}", old, &new)), entry)); } } Value::Uint64(old) => { if let Value::Uint64(new) = attempt.value { return Ok(write(Value::Uint64(sum([*old, new])), entry)); } } Value::Int64(old) => { if let Value::Int64(new) = attempt.value { return Ok(write(Value::Int64(sum([*old, new])), entry)); } } // Note: only works on the uint64 range in practice. Value::Uint128(old) => { if let Value::Uint128(new) = attempt.value { return Ok(write(Value::Uint128([0, sum([old[1], new[1]])]), entry)); } } _ => { panic!("actively unsupported type!") } } // Only reachable if the type of the would be concatenated value did not match the // value already occupying this entry. println!("Write error: INVALID_VALUE for key: {}", entry.key()); return Err(WriteError::InvalidValue); } // If we're not doing CONCAT, check for OVERWRITE next. if options.contains(WriteOptions::OVERWRITE) { return Ok(write(attempt.value, entry)); } println!("Write error: ALREADY_EXISTS for key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote key: {}, value: {:?}", entry.key(), &attempt.value); entry.insert(attempt.value.clone()); Ok(attempt.value) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, Value>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { // Because we are using a table payload, there is an extra level of indirection. The // top-level container for the table itself is always called "payload". StoreRequest::WriteItem { payload, responder } => { println!("WriteItem request received"); // Error out if either of the request table's members are not set. let attempt = payload.attempt.context("required field 'attempt' is unset")?; let options = payload.options.context("required field 'options' is unset")?; // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send( write_item(&mut store.borrow_mut(), attempt, &options) .as_ref() .map_err(|e| *e), ) .context("error sending reply")?; println!("WriteItem response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Size constraint
FIDL recipe: Size constraint
FIDL vectors and strings may carry a size constraint, which specifies a limit to how many members the type can contain. In the case of vectors this refers to the number of elements stored in the vector, while for strings it refers to the number-of-bytes the string contains.
The use of size constraints is strongly encouraged, since it sets an upper bound on what would otherwise be an unboundedly large type.
A useful operation for key-value stores is in-order iteration: that is, when given a key, to return a (usually paginated) list of elements that appear after it, in order.
Reasoning
In FIDL, this is best done using an iterator, which is generally implemented as a separate protocol over which this iteration can occur. Using a separate protocol, and therefore a separate channel, has a number of benefits, including de-interleaving the iteration pull requests from other operations done over the main protocol.
The client and server side of the channel connection for protocol P can be
represented as FIDL data types, as a client_end:P and server_end:P,
respectively. These types are collectively known as protocol ends, and
represent the other (non-@discoverable) way of connecting a FIDL client to its
corresponding server: over an existing FIDL connection!
Protocol ends are specific instances of a general FIDL concept: the resource type. A resource type is intended to contain FIDL handles, which necessitates extra restrictions on how the type can be used. The type must be always be unique, as the underlying resource is mediated by some other capability manager (usually the Zircon kernel). Duplicating such a resource via a simple in-memory copy, without involving the manager, is impossible. To prevent such duplication, all resource types in FIDL are always move-only.
Finally, the Get() method of the Iterator protocol itself makes use of a
size constraint on the return payload. This limits the amount of data that may
be transmitted in a single pull, allowing for some measure of resource use
control. It also creates a natural pagination boundary: rather than a giant dump
of all of the results at once, the server only needs to prepare small batches at
a time.
Implementation
The FIDL, CML, and realm interface definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.additerator; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value vector<byte>:64000; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// An enumeration of things that may go wrong when trying to create an iterator. type IterateConnectionError = flexible enum { /// The starting key was not found. UNKNOWN_START_AT = 1; }; /// A key-value store which supports insertion and iteration. @discoverable open protocol Store { /// Writes an item to the store. flexible WriteItem(struct { attempt Item; }) -> () error WriteError; /// Iterates over the items in the store, using lexicographic ordering over the keys. /// /// The [`iterator`] is [pipelined][pipelining] to the server, such that the client can /// immediately send requests over the new connection. /// /// [pipelining]: https://fuchsia.dev/fuchsia-src/development/api/fidl?hl=en#request-pipelining flexible Iterate(resource struct { /// If present, requests to start the iteration at this item. starting_at string:<128, optional>; /// The [`Iterator`] server endpoint. The client creates both ends of the channel and /// retains the `client_end` locally to use for pulling iteration pages, while sending the /// `server_end` off to be fulfilled by the server. iterator server_end:Iterator; }) -> () error IterateConnectionError; }; /// An iterator for the key-value store. Note that this protocol makes no guarantee of atomicity - /// the values may change between pulls from the iterator. Unlike the `Store` protocol above, this /// protocol is not `@discoverable`: it is not independently published by the component that /// implements it, but rather must have one of its two protocol ends transmitted over an existing /// FIDL connection. /// /// As is often the case with iterators, the client indicates that they are done with an instance of /// the iterator by simply closing their end of the connection. /// /// Since the iterator is associated only with the Iterate method, it is declared as closed rather /// than open. This is because changes to how iteration works are more likely to require replacing /// the Iterate method completely (which is fine because that method is flexible) rather than /// evolving the Iterator protocol. closed protocol Iterator { /// Gets the next batch of keys. /// /// The client pulls keys rather than having the server proactively push them, to implement /// [flow control][flow-control] over the messages. /// /// [flow-control]: /// https://fuchsia.dev/fuchsia-src/development/api/fidl?hl=en#prefer_pull_to_push strict Get() -> (struct { /// A list of keys. If the iterator has reached the end of iteration, the list will be /// empty. The client is expected to then close the connection. entries vector<string:128>:10; }); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.additerator.Store" }, ], config: { write_items: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, // A key to iterate from, after all items in `write_items` have been written. iterate_from: { type: "string", max_size: 64, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.additerator.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.additerator.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.additerator.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use config::Config; use fuchsia_component::client::connect_to_protocol; use std::{thread, time}; use fidl::endpoints::create_proxy; use fidl_examples_keyvaluestore_additerator::{Item, IteratorMarker, StoreMarker}; use futures::join; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // This client's structured config has one parameter, a vector of strings. Each string is the // path to a resource file whose filename is a key and whose contents are a value. We iterate // over them and try to write each key-value pair to the remote store. for key in config.write_items.into_iter() { let path = format!("/pkg/data/{}.txt", key); let value = std::fs::read_to_string(path.clone()) .with_context(|| format!("Failed to load {path}"))?; match store.write_item(&Item { key: key, value: value.into_bytes() }).await? { Ok(_) => println!("WriteItem Success"), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } } if !config.iterate_from.is_empty() { // This helper creates a channel, and returns two protocol ends: the `client_end` is already // conveniently bound to the correct FIDL protocol, `Iterator`, while the `server_end` is // unbound and ready to be sent over the wire. let (iterator, server_end) = create_proxy::<IteratorMarker>(); // There is no need to wait for the iterator to connect before sending the first `Get()` // request - since we already hold the `client_end` of the connection, we can start queuing // requests on it immediately. let connect_to_iterator = store.iterate(Some(config.iterate_from.as_str()), server_end); let first_get = iterator.get(); // Wait until both the connection and the first request resolve - an error in either case // triggers an immediate resolution of the combined future. let (connection, first_page) = join!(connect_to_iterator, first_get); // Handle any connection error. If this has occurred, it is impossible for the first `Get()` // call to have resolved successfully, so check this error first. if let Err(err) = connection.context("Could not connect to Iterator")? { println!("Iterator Connection Error: {}", err.into_primitive()); } else { println!("Iterator Connection Success"); // Consecutively repeat the `Get()` request if the previous response was not empty. let mut entries = first_page.context("Could not get page from Iterator")?; while !&entries.is_empty() { for entry in entries.iter() { println!("Iterator Entry: {}", entry); } entries = iterator.get().await.context("Could not get page from Iterator")?; } } } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fuchsia_component::server::ServiceFs; use futures::prelude::*; use lazy_static::lazy_static; use regex::Regex; use fidl_examples_keyvaluestore_additerator::{ Item, IterateConnectionError, IteratorRequest, IteratorRequestStream, StoreRequest, StoreRequestStream, WriteError, }; use fuchsia_async as fasync; use std::collections::btree_map::Entry; use std::collections::BTreeMap; use std::ops::Bound::*; use std::sync::{Arc, Mutex}; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } /// Handler for the `WriteItem` method. fn write_item(store: &mut BTreeMap<String, Vec<u8>>, attempt: Item) -> Result<(), WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY, For key: {}", attempt.key); return Err(WriteError::InvalidKey); } // Validate the value. if attempt.value.is_empty() { println!("Write error: INVALID_VALUE, For key: {}", attempt.key); return Err(WriteError::InvalidValue); } // Write to the store, validating that the key did not already exist. match store.entry(attempt.key) { Entry::Occupied(entry) => { println!("Write error: ALREADY_EXISTS, For key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote value at key: {}", entry.key()); entry.insert(attempt.value); Ok(()) } } } /// Handler for the `Iterate` method, which deals with validating that the requested start position /// exists, and then sets up the asynchronous side channel for the actual iteration to occur over. fn iterate( store: Arc<Mutex<BTreeMap<String, Vec<u8>>>>, starting_at: Option<String>, stream: IteratorRequestStream, ) -> Result<(), IterateConnectionError> { // Validate that the starting key, if supplied, actually exists. if let Some(start_key) = starting_at.clone() { if !store.lock().unwrap().contains_key(&start_key) { return Err(IterateConnectionError::UnknownStartAt); } } // Spawn a detached task. This allows the method call to return while the iteration continues in // a separate, unawaited task. fasync::Task::spawn(async move { // Serve the iteration requests. Note that access to the underlying store is behind a // contended `Mutex`, meaning that the iteration is not atomic: page contents could shift, // change, or disappear entirely between `Get()` requests. stream .map(|result| result.context("failed request")) .try_fold( match starting_at { Some(start_key) => Included(start_key), None => Unbounded, }, |mut lower_bound, request| async { match request { IteratorRequest::Get { responder } => { println!("Iterator page request received"); // The `page_size` should be kept in sync with the size constraint on // the iterator's response, as defined in the FIDL protocol. static PAGE_SIZE: usize = 10; // An iterator, beginning at `lower_bound` and tracking the pagination's // progress through iteration as each page is pulled by a client-sent // `Get()` request. let held_store = store.lock().unwrap(); let mut entries = held_store.range((lower_bound.clone(), Unbounded)); let mut current_page = vec![]; for _ in 0..PAGE_SIZE { match entries.next() { Some(entry) => { current_page.push(entry.0.clone()); } None => break, } } // Update the `lower_bound` - either inclusive of the next item in the // iteration, or exclusive of the last seen item if the iteration has // finished. This `lower_bound` will be passed to the next request // handler as its starting point. lower_bound = match entries.next() { Some(next) => Included(next.0.clone()), None => match current_page.last() { Some(tail) => Excluded(tail.clone()), None => lower_bound, }, }; // Send the page. At the end of this scope, the `held_store` lock gets // dropped, and therefore released. responder.send(¤t_page).context("error sending reply")?; println!("Iterator page sent"); } } Ok(lower_bound) }, ) .await .ok(); }) .detach(); Ok(()) } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. // // Note that we now use an `Arc<Mutex<BTreeMap>>`, replacing the previous `RefCell<HashMap>`. // The `BTreeMap` is used because we want an ordered map, to better facilitate iteration. The // `Arc<Mutex<...>>` is used because there are now multiple async tasks accessing the: one main // task which handles communication over the protocol, and one additional task per iterator // protocol. `Arc<Mutex<...>>` is the simplest way to synchronize concurrent access between // these racing tasks. let store = &Arc::new(Mutex::new(BTreeMap::<String, Vec<u8>>::new())); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { StoreRequest::WriteItem { attempt, responder } => { println!("WriteItem request received"); // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send(write_item(&mut store.clone().lock().unwrap(), attempt)) .context("error sending reply")?; println!("WriteItem response sent"); } StoreRequest::Iterate { starting_at, iterator, responder } => { println!("Iterate request received"); // The `iterate` handler does a quick check to see that the request is valid, // then spins up a separate worker task to serve the newly minted `Iterator` // protocol instance, allowing this call to return immediately and continue the // request stream with other work. responder .send(iterate(store.clone(), starting_at, iterator.into_stream())) .context("error sending reply")?; println!("Iterate response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Struct payload
FIDL recipe: Struct payload
A struct payload is a FIDL method payload that uses the
struct layout. The struct is a simple sequence of typed fields, similar to
how C structs work.
In this example, you will create a basic calculator server & client which shows the fundamental setup needed to first define and then serve and consume a FIDL protocol.
First, you will define the interface definitions and test harness. The
interface definition (the .fidl file itself) is the starting point for any new
FIDL protocol. Additionally, the calculator includes the necessary CML and realm
definitions to create a client-server pattern which can be used as project
scaffolding for arbitrary implementations.
See below for the FIDL code:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. // The namespace for this FIDL protocol. This namespace is how both consumers (clients) and providers (servers) reference this protocol. library examples.calculator.baseline; // @discoverable indicates 'Calculator' is a protocol that will be served under the examples.calculator.baseline libarary namespace. https://fuchsia.dev/fuchsia-src/reference/fidl/language/attributes#discoverable . If @discoverable is missing, it will lead to a compile time error when trying to import the library. @discoverable // A limited-functionality calculator 'protocol' that adds and subtracts integers. open protocol Calculator { // Takes as input a struct with two integers, and returns their sum: (a+b)=sum. This method is infallible (no errors can be generated) as two int32's cannot overflow a result type of int64. flexible Add(struct { a int32; b int32; }) -> (struct { sum int64; }); // Takes as input a struct with two integers, and returns their difference: (a-b)=difference. This method is infallible (no errors can be generated) as two int32's cannot overflow a result type of int64. flexible Subtract(struct { a int32; b int32; }) -> (struct { difference int64; }); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.calculator.baseline.Calculator" }, ], config: {}, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.calculator.baseline.Calculator" }, ], expose: [ { protocol: "examples.calculator.baseline.Calculator", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.calculator.baseline.Calculator", from: "#server", to: "#client", }, // Route logging support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// TODO(https://fxbug.dev/42063075): Rust implementation.Server
// TODO(https://fxbug.dev/42063075): Rust implementation.C++ (Natural)
Client
// TODO(https://fxbug.dev/42063075): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42063075): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42063075): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42063075): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42063075): HLCPP implementation.Server
// TODO(https://fxbug.dev/42063075): HLCPP implementation.Creating a FIDL protocol from the ground up as is shown in this example can be a more common scenario for certain developers, such as platform developers. However, other types of developers also benefit from learning how to construct a FIDL protocol even if they won't typically do so. This helps you learn how everything about FIDL fits together, including the syntax, grammar, language features, how to serve and consume a given FIDL protocol, and how the build system works. For next steps, the examples which follow this baseline show how to extend an existing FIDL protocol, which is expected to be a fairly common practice.
Table payload
FIDL recipe: Table payload
A table payload is a FIDL method payload that uses the table layout. The
top-level type used as the method payload must use one of struct, table, or
union as its layout. Notably, some generated bindings "flatten" the arguments
passed to struct method payloads, such that each member is itself treated as a
function argument in the calling signature. Payloads that use table or union
never do this, and always pass a single argument, called payload, instead.
Reasoning
The key-value store baseline example's implementation was a good starting point, but one major drawback is that data is stored as raw bytes. FIDL is a richly typed language. Forcing data that is for instance a UTF-8 string to be stored as an untyped byte array erases this valuable type information for readers of the *.fidl file, as well as for programmers using bindings generated from it.
Implementation
The main goal of this change is to replace the baseline case's vector<byte>
typed value member with a union that stores many possible types. In fact, as
of this change a good survey of FIDL's
value types is on
offer:
- All of FIDL's builtin scalar types are used as variants in the
Valueunion:bool,uint8,uint16,uint32,uint64,int8,int16,int32,int64,float32, andfloat64(also known as FIDL's primitive types), as well asstring. - This
unionalso features uses of FIDL's builtinarray<T, N>andvector<T>type templates. - All of FIDL's type layouts, namely
bits,enum,table,union, andstruct, are utilized in this example at least once.
The request and response payloads used for WriteItem have also been changed
from structs to a named table and an inlined flexible union, respectively.
In fact, any of these three layouts may be used a request/response payload. The
latter two, known as table payloads and *union payloads, respectively, are
preferred in all but the most message size sensitive cases. This is because they
are much easier to extend in the future in a binary compatible way.
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.usegenericvalues; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value Value; }; // Because the `Value` must be used both in the request and the response, we give it its own named // type. The type is a `union` of all possible data types that we take as values, and is marked // `flexible` to allow for the easy addition of new data types in the future. type Value = flexible union { // Keep the original `bytes` as one of the options in the new union. 1: bytes vector<byte>:64000; // A `string` is very similar to `vector<byte>` on the wire, with the extra constraint that // it enforces that it enforces that the byte vector in question is valid UTF-8. 2: string string:64000; // All of FIDL's primitive types. 3: bool bool; 4: uint8 uint8; 5: int8 int8; 6: uint16 uint16; 7: int16 int16; 8: uint32 uint32; 9: int32 int32; 10: float32 float32; 11: uint64 uint64; 12: int64 int64; 13: float64 float64; // FIDL does not natively support 128-bit integer types, so we have to define our own // representations. 14: uint128 array<uint64, 2>; }; // Because we now supoprt a richer range of types as values in our store, it is helpful to use a // `flexible`, and therefore evolvable, `bits` type to store write options. type WriteOptions = flexible bits : uint8 { // This flag allows us to overwrite existing data when there is a collision, rather than failing // with an `WriteError.ALREADY_EXISTS`. OVERWRITE = 0b1; // This flag allows us to concatenate to existing data when there is a collision, rather than // failing with an `WriteError.ALREADY_EXISTS`. "Concatenation" means addition for the numeric // variants and appending to the `bytes`/`string` variants. If no existing data can be found, we // "concatenate" to default values of zero and an empty vector, respectively. Attempting to // concatenate to an existing variant of a different type will return a // `WriteError.INVALID_VALUE` error. CONCAT = 0b10; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// A very basic key-value store. @discoverable open protocol Store { /// Writes an item to the store. /// /// Since the value stored in the key-value store can now be different from the input (if the /// `WriteOptions.CONCAT` flag is set), we need to return the resulting `Value` to the /// requester. /// /// We use an (anonymous) `table` and a (named) `flexible union` as the request and response /// payload, respectively, to allow for easier future evolution. Both of these types are /// `flexible`, meaning that adding or removing members is binary-compatible. This makes them /// much easier to evolve that the `struct` types that were previously used, which cannot be /// changed after release without breaking ABI. flexible WriteItem(table { 1: attempt Item; 2: options WriteOptions; }) -> (Value) error WriteError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store" }, ], config: { // A vector of values for every easily representible type in our key-value store. For // brevity's sake, the 8, 16, and 32 bit integer types and booleans are omitted. // // TODO(https://fxbug.dev/42178362): It would absolve individual language implementations of a great // deal of string parsing if we were able to use all FIDL constructs directly here. In // particular, floats and nested types are very difficult to represent, and have been // excluded from this example for the time being. set_concat_option: { type: "bool" }, set_overwrite_option: { type: "bool" }, write_bytes: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, write_strings: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, write_uint64s: { type: "vector", max_count: 16, element: { type: "uint64" }, }, write_int64s: { type: "vector", max_count: 16, element: { type: "int64" }, }, // Note: due to the limitation of structured config not allowing vectors nested in vectors, // we only set the lower half of the uint128 for simplicity's sake. write_uint128s: { type: "vector", max_count: 16, element: { type: "uint64" }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.usegenericvalues.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, config::Config, fidl_examples_keyvaluestore_usegenericvalues::{ Item, StoreMarker, StoreProxy, StoreWriteItemRequest, Value, WriteOptions, }, fuchsia_component::client::connect_to_protocol, std::{thread, time}, }; // A helper function to sequentially write a single item to the key-value store and print a log when // successful. async fn write_next_item( store: &StoreProxy, key: &str, value: Value, options: WriteOptions, ) -> Result<(), Error> { // Create an empty request payload using `::default()`. let mut req = StoreWriteItemRequest::default(); req.options = Some(options); // Fill in the `Item` we will be attempting to write. println!("WriteItem request sent: key: {}, value: {:?}", &key, &value); req.attempt = Some(Item { key: key.to_string(), value: value }); // Send and async `WriteItem` request to the server. match store.write_item(&req).await.context("Error sending request")? { Ok(value) => println!("WriteItem response received: {:?}", &value), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } Ok(()) } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // All of our requests will have the same bitflags set. Pull these settings from the config. let mut options = WriteOptions::empty(); options.set(WriteOptions::OVERWRITE, config.set_overwrite_option); options.set(WriteOptions::CONCAT, config.set_concat_option); // The structured config provides one input for most data types that can be stored in the data // store. Iterate through those inputs in the order we see them in the FIDL file. // // Note that FIDL unions are rendered as enums in Rust; for example, the `Value` union has now // become a `Value` Rust enum, with each member taking exactly one argument. for value in config.write_bytes.into_iter() { write_next_item(&store, "bytes", Value::Bytes(value.into()), options).await?; } for value in config.write_strings.into_iter() { write_next_item(&store, "string", Value::String(value), options).await?; } for value in config.write_uint64s.into_iter() { write_next_item(&store, "uint64", Value::Uint64(value), options).await?; } for value in config.write_int64s.into_iter() { write_next_item(&store, "int64", Value::Int64(value), options).await?; } for value in config.write_uint128s.into_iter() { write_next_item(&store, "uint128", Value::Uint128([0, value]), options).await?; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fuchsia_component::server::ServiceFs; use futures::prelude::*; use lazy_static::lazy_static; use regex::Regex; use std::cell::RefCell; use std::collections::hash_map::Entry; use std::collections::HashMap; use fidl_examples_keyvaluestore_usegenericvalues::{ Item, StoreRequest, StoreRequestStream, Value, WriteError, WriteOptions, }; use std::collections::hash_map::OccupiedEntry; use std::ops::Add; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } /// Sums any numeric type. fn sum<T: Add + Add<Output = T> + Copy>(operands: [T; 2]) -> T { operands[0] + operands[1] } /// Clones and inserts an entry, so that the original (now concatenated) copy may be returned in the /// response. fn write(inserting: Value, mut entry: OccupiedEntry<'_, String, Value>) -> Value { entry.insert(inserting.clone()); println!("Wrote key: {}, value: {:?}", entry.key(), &inserting); inserting } /// Handler for the `WriteItem` method. fn write_item( store: &mut HashMap<String, Value>, attempt: Item, options: &WriteOptions, ) -> Result<Value, WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY for key: {}", attempt.key); return Err(WriteError::InvalidKey); } match store.entry(attempt.key) { Entry::Occupied(entry) => { // The `CONCAT` flag supersedes the `OVERWRITE` flag, so check it first. if options.contains(WriteOptions::CONCAT) { match entry.get() { Value::Bytes(old) => { if let Value::Bytes(new) = attempt.value { let mut combined = old.clone(); combined.extend(new); return Ok(write(Value::Bytes(combined), entry)); } } Value::String(old) => { if let Value::String(new) = attempt.value { return Ok(write(Value::String(format!("{}{}", old, &new)), entry)); } } Value::Uint64(old) => { if let Value::Uint64(new) = attempt.value { return Ok(write(Value::Uint64(sum([*old, new])), entry)); } } Value::Int64(old) => { if let Value::Int64(new) = attempt.value { return Ok(write(Value::Int64(sum([*old, new])), entry)); } } // Note: only works on the uint64 range in practice. Value::Uint128(old) => { if let Value::Uint128(new) = attempt.value { return Ok(write(Value::Uint128([0, sum([old[1], new[1]])]), entry)); } } _ => { panic!("actively unsupported type!") } } // Only reachable if the type of the would be concatenated value did not match the // value already occupying this entry. println!("Write error: INVALID_VALUE for key: {}", entry.key()); return Err(WriteError::InvalidValue); } // If we're not doing CONCAT, check for OVERWRITE next. if options.contains(WriteOptions::OVERWRITE) { return Ok(write(attempt.value, entry)); } println!("Write error: ALREADY_EXISTS for key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote key: {}, value: {:?}", entry.key(), &attempt.value); entry.insert(attempt.value.clone()); Ok(attempt.value) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, Value>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { // Because we are using a table payload, there is an extra level of indirection. The // top-level container for the table itself is always called "payload". StoreRequest::WriteItem { payload, responder } => { println!("WriteItem request received"); // Error out if either of the request table's members are not set. let attempt = payload.attempt.context("required field 'attempt' is unset")?; let options = payload.options.context("required field 'options' is unset")?; // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send( write_item(&mut store.borrow_mut(), attempt, &options) .as_ref() .map_err(|e| *e), ) .context("error sending reply")?; println!("WriteItem response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Throttled event pattern
FIDL recipe: Throttled event pattern
Events are FIDL calls initiated from the server. Because these calls have no built-in client-side response, they are not flow controlled: a server could queue up a very large number of such calls, and flood the client. One solution to this problem is the throttled event pattern. This pattern involves adding a client-called FIDL method to serve as an acknowledgement point for one or more events to sync to.
Servers should refrain from sending more of the throttled events (the exact semantics here are specific to the implementing protocol) until they have received the next acknowledgement call from the client. Similarly, clients should close the connection the if servers send more throttled events than allowed before the client has acknowledged them. These restrictions are not built into the FIDL runtime, and require some manual implementation on the part of client/server implementers to ensure correct behavior.
One way to improve the performance of the Instance protocol is to allow the
batching of lines: rather than sending a single AddLine(...); every time we
have a new line we'd like to add to the canvas, waiting for the reply, then
doing it again for the next line, we can instead batch many lines into a single
invocation of the new AddLines(...); call. The client can now decide how to
best segment a large collection of lines to be drawn.
Naively implemented, we would find ourselves in a situation where the server and
the client are completely unsynchronized: the client can flood the server with
unbounded AddLines(...); calls, and the server can similarly flood the client
with more -> OnDrawn(...); events than it can handle. The solution to both of
these problems is to add a simple Ready() -> (); method for synchronization
purposes. This method is called by the client whenever it is prepared to receive
the next draw update, with the response from the server indicating that the client
can proceed with more requests.
We now have some flow control in both directions. The protocol now implements
the feed forward pattern, allowing many uncontrolled calls before some
synchronizing "commit" call triggers the actual work on the server. This
prevents the client from overwhelming the server with work. Similarly, the
server is no longer allowed to send unbounded -> OnDrawn(...); events: each
event must follow a signal from the client, the Ready() -> (); call, that
indicates that it is ready to do more work. This is known as the throttled
event pattern.
The concrete implementations must apply some of these rules manually: the client
must close the connection if it receives an -> OnDrawn(...); event it did not
request via the Ready() -> (); method.
The FIDL, CML, and realm interface definitions are as follows:
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.canvas.clientrequesteddraw; /// A point in 2D space. type Point = struct { x int64; y int64; }; /// A line in 2D space. alias Line = array<Point, 2>; /// A bounding box in 2D space. This is the result of "drawing" operations on our canvas, and what /// the server reports back to the client. These bounds are sufficient to contain all of the /// lines (inclusive) on a canvas at a given time. type BoundingBox = struct { top_left Point; bottom_right Point; }; /// Manages a single instance of a canvas. Each session of this protocol is responsible for a new /// canvas. @discoverable open protocol Instance { /// Add multiple lines to the canvas. We are able to reduce protocol chatter and the number of /// requests needed by batching instead of calling the simpler `AddLine(...)` one line at a /// time. flexible AddLines(struct { lines vector<Line>; }); /// Rather than the server randomly performing draws, or trying to guess when to do so, the /// client must explicitly ask for them. This creates a bit of extra chatter with the additional /// method invocation, but allows much greater client-side control of when the canvas is "ready" /// for a view update, thereby eliminating unnecessary draws. /// /// This method also has the benefit of "throttling" the `-> OnDrawn(...)` event - rather than /// allowing a potentially unlimited flood of `-> OnDrawn(...)` calls, we now have the runtime /// enforced semantic that each `-> OnDrawn(...)` call must follow a unique `Ready() -> ()` call /// from the client. An unprompted `-> OnDrawn(...)` is invalid, and should cause the channel to /// immediately close. flexible Ready() -> (); /// Update the client with the latest drawing state. The server makes no guarantees about how /// often this event occurs - it could occur multiple times per board state, for example. flexible -> OnDrawn(BoundingBox); };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.canvas.clientrequesteddraw.Instance" }, ], config: { // A script for the client to follow. Entries in the script may take one of two forms: a // pair of signed-integer coordinates like "-2,15:4,5", or the string "READY". The former // builds a local vector sent via a single `AddLines(...)` call, while the latter sends a // `Ready() -> ()` call pauses execution until the next `->OnDrawn(...)` event is received. // // TODO(https://fxbug.dev/42178362): It would absolve individual language implementations of a great // deal of string parsing if we were able to use a vector of `union { Point; Ready}` here. script: { type: "vector", max_count: 100, element: { type: "string", max_size: 64, }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.canvas.clientrequesteddraw.Instance" }, ], expose: [ { protocol: "examples.canvas.clientrequesteddraw.Instance", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.canvas.clientrequesteddraw.Instance", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{format_err, Context as _, Error}; use config::Config; use fidl_examples_canvas_clientrequesteddraw::{InstanceEvent, InstanceMarker, Point}; use fuchsia_component::client::connect_to_protocol; use futures::TryStreamExt; use std::{thread, time}; #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send Instance requests // across the channel. let instance = connect_to_protocol::<InstanceMarker>()?; println!("Outgoing connection enabled"); let mut batched_lines = Vec::<[Point; 2]>::new(); for action in config.script.into_iter() { // If the next action in the script is to "PUSH", send a batch of lines to the server. if action == "PUSH" { instance.add_lines(&batched_lines).context("Could not send lines")?; println!("AddLines request sent"); batched_lines.clear(); continue; } // If the next action in the script is to "WAIT", block until an OnDrawn event is received // from the server. if action == "WAIT" { let mut event_stream = instance.take_event_stream(); loop { match event_stream .try_next() .await .context("Error getting event response from proxy")? .ok_or_else(|| format_err!("Proxy sent no events"))? { InstanceEvent::OnDrawn { top_left, bottom_right } => { println!( "OnDrawn event received: top_left: {:?}, bottom_right: {:?}", top_left, bottom_right ); break; } InstanceEvent::_UnknownEvent { ordinal, .. } => { println!("Received an unknown event with ordinal {ordinal}"); } } } // Now, inform the server that we are ready to receive more updates whenever they are // ready for us. println!("Ready request sent"); instance.ready().await.context("Could not send ready call")?; println!("Ready success"); continue; } // Add a line to the next batch. Parse the string input, making two points out of it. let mut points = action .split(":") .map(|point| { let integers = point .split(",") .map(|integer| integer.parse::<i64>().unwrap()) .collect::<Vec<i64>>(); Point { x: integers[0], y: integers[1] } }) .collect::<Vec<Point>>(); // Assemble a line from the two points. let from = points.pop().ok_or_else(|| format_err!("line requires 2 points, but has 0"))?; let to = points.pop().ok_or_else(|| format_err!("line requires 2 points, but has 1"))?; let mut line: [Point; 2] = [from, to]; // Batch a line for drawing to the canvas using the two points provided. println!("AddLines batching line: {:?}", &mut line); batched_lines.push(line); } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{anyhow, Context as _, Error}; use fidl::endpoints::RequestStream as _; use fidl_examples_canvas_clientrequesteddraw::{ BoundingBox, InstanceRequest, InstanceRequestStream, Point, }; use fuchsia_async::{MonotonicInstant, Timer}; use fuchsia_component::server::ServiceFs; use futures::future::join; use futures::prelude::*; use std::sync::{Arc, Mutex}; // A struct that stores the two things we care about for this example: the bounding box the lines // that have been added thus far, and bit to track whether or not there have been changes since the // last `OnDrawn` event. #[derive(Debug)] struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. changed: bool, // Tracks whether or not the client has declared itself ready to receive more updated. ready: bool, bounding_box: BoundingBox, } /// Handler for the `AddLines` method. fn add_lines(state: &mut CanvasState, lines: Vec<[Point; 2]>) { // Update the bounding box to account for the new lines we've just "added" to the canvas. let bounds = &mut state.bounding_box; for line in lines { println!("AddLines printing line: {:?}", line); for point in line { if point.x < bounds.top_left.x { bounds.top_left.x = point.x; } if point.y > bounds.top_left.y { bounds.top_left.y = point.y; } if point.x > bounds.bottom_right.x { bounds.bottom_right.x = point.x; } if point.y < bounds.bottom_right.y { bounds.bottom_right.y = point.y; } } } // Mark the state as "dirty", so that an update is sent back to the client on the next tick. state.changed = true } /// Creates a new instance of the server, paired to a single client across a zircon channel. async fn run_server(stream: InstanceRequestStream) -> Result<(), Error> { // Create a new in-memory state store for the state of the canvas. The store will live for the // lifetime of the connection between the server and this particular client. let state = Arc::new(Mutex::new(CanvasState { changed: true, ready: true, bounding_box: BoundingBox { top_left: Point { x: 0, y: 0 }, bottom_right: Point { x: 0, y: 0 }, }, })); // Take ownership of the control_handle from the stream, which will allow us to push events from // a different async task. let control_handle = stream.control_handle(); // A separate watcher task periodically "draws" the canvas, and notifies the client of the new // state. We'll need a cloned reference to the canvas state to be accessible from the new // task. let state_ref = state.clone(); let update_sender = || async move { loop { // Our server sends one update per second, but only if the client has declared that it // is ready to receive one. Timer::new(MonotonicInstant::after(zx::MonotonicDuration::from_seconds(1))).await; let mut state = state_ref.lock().unwrap(); if !state.changed || !state.ready { continue; } // After acquiring the lock, this is where we would draw the actual lines. Since this is // just an example, we'll avoid doing the actual rendering, and simply send the bounding // box to the client instead. let bounds = state.bounding_box; match control_handle.send_on_drawn(&bounds.top_left, &bounds.bottom_right) { Ok(_) => println!( "OnDrawn event sent: top_left: {:?}, bottom_right: {:?}", bounds.top_left, bounds.bottom_right ), Err(_) => return, } // Reset the change and ready trackers. state.ready = false; state.changed = false; } }; // Handle requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. let state_ref = &state; let request_handler = stream.map(|result| result.context("failed request")).try_for_each(|request| async move { // Match based on the method being invoked. match request { InstanceRequest::AddLines { lines, .. } => { println!("AddLines request received"); add_lines(&mut state_ref.lock().unwrap(), lines); } InstanceRequest::Ready { responder, .. } => { println!("Ready request received"); // The client must only call `Ready() -> ();` after receiving an `-> OnDrawn();` // event; if two "consecutive" `Ready() -> ();` calls are received, this // interaction has entered an invalid state, and should be aborted immediately. let mut state = state_ref.lock().unwrap(); if state.ready == true { return Err(anyhow!("Invalid back-to-back `Ready` requests received")); } state.ready = true; responder.send().context("Error responding")?; } // InstanceRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }); // This line will only be reached if the server errors out. The stream will await indefinitely, // thereby creating a long-lived server. Here, we first wait for the updater task to realize the // connection has died, then bubble up the error. join(request_handler, update_sender()).await.0 } // A helper enum that allows us to treat a `Instance` service instance as a value. enum IncomingService { Instance(InstanceRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Instance` protocol - this will allow the client to see // the server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Instance); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Instance(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.clientrequesteddraw/cpp/fidl.h> #include <lib/async-loop/cpp/loop.h> #include <lib/component/incoming/cpp/protocol.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <charconv> #include <examples/fidl/new/canvas/client_requested_draw/cpp_natural/client/config.h> // The |EventHandler| is a derived class that we pass into the |fidl::WireClient| to handle incoming // events asynchronously. class EventHandler : public fidl::AsyncEventHandler<examples_canvas_clientrequesteddraw::Instance> { public: // Handler for |OnDrawn| events sent from the server. void OnDrawn( fidl::Event<examples_canvas_clientrequesteddraw::Instance::OnDrawn>& event) override { ::examples_canvas_clientrequesteddraw::Point top_left = event.top_left(); ::examples_canvas_clientrequesteddraw::Point bottom_right = event.bottom_right(); FX_LOGS(INFO) << "OnDrawn event received: top_left: Point { x: " << top_left.x() << ", y: " << top_left.y() << " }, bottom_right: Point { x: " << bottom_right.x() << ", y: " << bottom_right.y() << " }"; loop_.Quit(); } void on_fidl_error(fidl::UnbindInfo error) override { FX_LOGS(ERROR) << error; } void handle_unknown_event( fidl::UnknownEventMetadata<examples_canvas_clientrequesteddraw::Instance> metadata) override { FX_LOGS(WARNING) << "Received an unknown event with ordinal " << metadata.event_ordinal; } explicit EventHandler(async::Loop& loop) : loop_(loop) {} private: async::Loop& loop_; }; // A helper function that takes a coordinate in string form, like "123,-456", and parses it into a // a struct of the form |{ in64 x; int64 y; }|. ::examples_canvas_clientrequesteddraw::Point ParsePoint(std::string_view input) { int64_t x = 0; int64_t y = 0; size_t index = input.find(','); if (index != std::string::npos) { std::from_chars(input.data(), input.data() + index, x); std::from_chars(input.data() + index + 1, input.data() + input.length(), y); } return ::examples_canvas_clientrequesteddraw::Point(x, y); } using Line = ::std::array<::examples_canvas_clientrequesteddraw::Point, 2>; // A helper function that takes a coordinate pair in string form, like "1,2:-3,-4", and parses it // into an array of 2 |Point| structs. Line ParseLine(const std::string& action) { auto input = std::string_view(action); size_t index = input.find(':'); if (index != std::string::npos) { return {ParsePoint(input.substr(0, index)), ParsePoint(input.substr(index + 1))}; } return {}; } int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop and dispatcher. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace. This can fail so it's wrapped in a // |zx::result| and it must be checked for errors. zx::result client_end = component::Connect<examples_canvas_clientrequesteddraw::Instance>(); if (!client_end.is_ok()) { FX_LOGS(ERROR) << "Synchronous error when connecting to the |Instance| protocol: " << client_end.status_string(); return -1; } // Create an instance of the event handler. EventHandler event_handler(loop); // Create an asynchronous client using the newly-established connection. fidl::Client client(std::move(*client_end), dispatcher, &event_handler); FX_LOGS(INFO) << "Outgoing connection enabled"; std::vector<Line> batched_lines; for (const auto& action : conf.script()) { // If the next action in the script is to "PUSH", send a batch of lines to the server. if (action == "PUSH") { fit::result<fidl::Error> result = client->AddLines(batched_lines); if (!result.is_ok()) { // Check that our one-way call was enqueued successfully, and handle the error // appropriately. In the case of this example, there is nothing we can do to recover here, // except to log an error and exit the program. FX_LOGS(ERROR) << "Could not send AddLines request: " << result.error_value(); return -1; } batched_lines.clear(); FX_LOGS(INFO) << "AddLines request sent"; continue; } // If the next action in the script is to "WAIT", block until an |OnDrawn| event is received // from the server. if (action == "WAIT") { loop.Run(); loop.ResetQuit(); // Now, inform the server that we are ready to receive more updates whenever they are // ready for us. FX_LOGS(INFO) << "Ready request sent"; client->Ready().ThenExactlyOnce( [&](fidl::Result<examples_canvas_clientrequesteddraw::Instance::Ready> result) { // Check if the FIDL call succeeded or not. if (result.is_ok()) { FX_LOGS(INFO) << "Ready success"; } else { FX_LOGS(ERROR) << "Could not send Ready request: " << result.error_value(); } // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); continue; } // Batch a line for drawing to the canvas using the two points provided. Line line = ParseLine(action); batched_lines.push_back(line); FX_LOGS(INFO) << "AddLines batching line: [Point { x: " << line[1].x() << ", y: " << line[1].y() << " }, Point { x: " << line[0].x() << ", y: " << line[0].y() << " }]"; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.clientrequesteddraw/cpp/fidl.h> #include <lib/async-loop/cpp/loop.h> #include <lib/async/cpp/task.h> #include <lib/component/outgoing/cpp/outgoing_directory.h> #include <lib/fidl/cpp/wire/channel.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <src/lib/fxl/macros.h> #include <src/lib/fxl/memory/weak_ptr.h> // A struct that stores the two things we care about for this example: the set of lines, and the // bounding box that contains them. struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. bool changed = true; // Tracks whether or not the client has declared itself ready to receive more updated. bool ready = true; examples_canvas_clientrequesteddraw::BoundingBox bounding_box; }; // An implementation of the |Instance| protocol. class InstanceImpl final : public fidl::Server<examples_canvas_clientrequesteddraw::Instance> { public: // Bind this implementation to a channel. InstanceImpl(async_dispatcher_t* dispatcher, fidl::ServerEnd<examples_canvas_clientrequesteddraw::Instance> server_end) : binding_(dispatcher, std::move(server_end), this, std::mem_fn(&InstanceImpl::OnFidlClosed)), weak_factory_(this) { // Start the update timer on startup. Our server sends one update per second ScheduleOnDrawnEvent(dispatcher, zx::sec(1)); } void OnFidlClosed(fidl::UnbindInfo info) { if (info.reason() != ::fidl::Reason::kPeerClosedWhileReading) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; } void AddLines(AddLinesRequest& request, AddLinesCompleter::Sync& completer) override { FX_LOGS(INFO) << "AddLines request received"; for (const auto& points : request.lines()) { FX_LOGS(INFO) << "AddLines printing line: [Point { x: " << points[1].x() << ", y: " << points[1].y() << " }, Point { x: " << points[0].x() << ", y: " << points[0].y() << " }]"; // Update the bounding box to account for the new line we've just "added" to the canvas. auto& bounds = state_.bounding_box; for (const auto& point : points) { if (point.x() < bounds.top_left().x()) { bounds.top_left().x() = point.x(); } if (point.y() > bounds.top_left().y()) { bounds.top_left().y() = point.y(); } if (point.x() > bounds.bottom_right().x()) { bounds.bottom_right().x() = point.x(); } if (point.y() < bounds.bottom_right().y()) { bounds.bottom_right().y() = point.y(); } } } // Mark the state as "dirty", so that an update is sent back to the client on the next |OnDrawn| // event. state_.changed = true; } void Ready(ReadyCompleter::Sync& completer) override { FX_LOGS(INFO) << "Ready request received"; // The client must only call `Ready() -> ();` after receiving an `-> OnDrawn();` event; if two // "consecutive" `Ready() -> ();` calls are received, this interaction has entered an invalid // state, and should be aborted immediately. if (state_.ready == true) { FX_LOGS(ERROR) << "Invalid back-to-back `Ready` requests received"; } state_.ready = true; completer.Reply(); } void handle_unknown_method( fidl::UnknownMethodMetadata<examples_canvas_clientrequesteddraw::Instance> metadata, fidl::UnknownMethodCompleter::Sync& completer) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << metadata.method_ordinal; } private: // Each scheduled update waits for the allotted amount of time, sends an update if something has // changed, and schedules the next update. void ScheduleOnDrawnEvent(async_dispatcher_t* dispatcher, zx::duration after) { async::PostDelayedTask( dispatcher, [&, dispatcher, after, weak = weak_factory_.GetWeakPtr()] { // Halt execution if the binding has been deallocated already. if (!weak) { return; } // Schedule the next update if the binding still exists. weak->ScheduleOnDrawnEvent(dispatcher, after); // No need to send an update if nothing has changed since the last one, or the client has // not yet informed us that it is ready for more updates. if (!weak->state_.changed || !weak->state_.ready) { return; } // This is where we would draw the actual lines. Since this is just an example, we'll // avoid doing the actual rendering, and simply send the bounding box to the client // instead. auto result = fidl::SendEvent(binding_)->OnDrawn(state_.bounding_box); if (!result.is_ok()) { return; } auto top_left = state_.bounding_box.top_left(); auto bottom_right = state_.bounding_box.bottom_right(); FX_LOGS(INFO) << "OnDrawn event sent: top_left: Point { x: " << top_left.x() << ", y: " << top_left.y() << " }, bottom_right: Point { x: " << bottom_right.x() << ", y: " << bottom_right.y() << " }"; // Reset the change and ready trackers. state_.ready = false; state_.changed = false; }, after); } fidl::ServerBinding<examples_canvas_clientrequesteddraw::Instance> binding_; CanvasState state_ = CanvasState{}; // Generates weak references to this object, which are appropriate to pass into asynchronous // callbacks that need to access this object. The references are automatically invalidated // if this object is destroyed. fxl::WeakPtrFactory<InstanceImpl> weak_factory_; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. This // directory is where the outgoing FIDL protocols are installed so that they can be provided to // other components. component::OutgoingDirectory outgoing = component::OutgoingDirectory(dispatcher); // The `ServeFromStartupInfo()` function sets up the outgoing directory with the startup handle. // The startup handle is a handle provided to every component by the system, so that they can // serve capabilities (e.g. FIDL protocols) to other components. zx::result result = outgoing.ServeFromStartupInfo(); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to serve outgoing directory: " << result.status_string(); return -1; } // Register a handler for components trying to connect to // |examples.canvas.clientrequesteddraw.Instance|. result = outgoing.AddUnmanagedProtocol<examples_canvas_clientrequesteddraw::Instance>( [dispatcher](fidl::ServerEnd<examples_canvas_clientrequesteddraw::Instance> server_end) { // Create an instance of our InstanceImpl that destroys itself when the connection closes. new InstanceImpl(dispatcher, std::move(server_end)); }); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to add Instance protocol: " << result.status_string(); return -1; } // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
C++ (Wire)
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.clientrequesteddraw/cpp/wire.h> #include <lib/async-loop/cpp/loop.h> #include <lib/component/incoming/cpp/protocol.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <charconv> #include <examples/fidl/new/canvas/client_requested_draw/cpp_wire/client/config.h> // The |EventHandler| is a derived class that we pass into the |fidl::WireClient| to handle incoming // events asynchronously. class EventHandler : public fidl::WireAsyncEventHandler<examples_canvas_clientrequesteddraw::Instance> { public: // Handler for |OnDrawn| events sent from the server. void OnDrawn( fidl::WireEvent<examples_canvas_clientrequesteddraw::Instance::OnDrawn>* event) override { ::examples_canvas_clientrequesteddraw::wire::Point top_left = event->top_left; ::examples_canvas_clientrequesteddraw::wire::Point bottom_right = event->bottom_right; FX_LOGS(INFO) << "OnDrawn event received: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; loop_.Quit(); } void on_fidl_error(fidl::UnbindInfo error) override { FX_LOGS(ERROR) << error; } void handle_unknown_event( fidl::UnknownEventMetadata<examples_canvas_clientrequesteddraw::Instance> metadata) override { FX_LOGS(WARNING) << "Received an unknown event with ordinal " << metadata.event_ordinal; } explicit EventHandler(async::Loop& loop) : loop_(loop) {} private: async::Loop& loop_; }; // A helper function that takes a coordinate in string form, like "123,-456", and parses it into a // a struct of the form |{ in64 x; int64 y; }|. ::examples_canvas_clientrequesteddraw::wire::Point ParsePoint(std::string_view input) { int64_t x = 0; int64_t y = 0; size_t index = input.find(','); if (index != std::string::npos) { std::from_chars(input.data(), input.data() + index, x); std::from_chars(input.data() + index + 1, input.data() + input.length(), y); } return ::examples_canvas_clientrequesteddraw::wire::Point{.x = x, .y = y}; } using Line = ::fidl::Array<::examples_canvas_clientrequesteddraw::wire::Point, 2>; // A helper function that takes a coordinate pair in string form, like "1,2:-3,-4", and parses it // into an array of 2 |Point| structs. Line ParseLine(const std::string& action) { auto input = std::string_view(action); size_t index = input.find(':'); if (index != std::string::npos) { return {ParsePoint(input.substr(0, index)), ParsePoint(input.substr(index + 1))}; } return {}; } int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop and dispatcher. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace. This can fail so it's wrapped in a // |zx::result| and it must be checked for errors. zx::result client_end = component::Connect<examples_canvas_clientrequesteddraw::Instance>(); if (!client_end.is_ok()) { FX_LOGS(ERROR) << "Synchronous error when connecting to the |Instance| protocol: " << client_end.status_string(); return -1; } // Create an instance of the event handler. EventHandler event_handler(loop); // Create an asynchronous client using the newly-established connection. fidl::WireClient client(std::move(*client_end), dispatcher, &event_handler); FX_LOGS(INFO) << "Outgoing connection enabled"; std::vector<Line> batched_lines; for (const auto& action : conf.script()) { // If the next action in the script is to "PUSH", send a batch of lines to the server. if (action == "PUSH") { fidl::Status status = client->AddLines(fidl::VectorView<Line>::FromExternal(batched_lines)); if (!status.ok()) { // Check that our one-way call was enqueued successfully, and handle the error // appropriately. In the case of this example, there is nothing we can do to recover here, // except to log an error and exit the program. FX_LOGS(ERROR) << "Could not send AddLines request: " << status.error(); return -1; } batched_lines.clear(); FX_LOGS(INFO) << "AddLines request sent"; continue; } // If the next action in the script is to "WAIT", block until an |OnDrawn| event is received // from the server. if (action == "WAIT") { loop.Run(); loop.ResetQuit(); // Now, inform the server that we are ready to receive more updates whenever they are // ready for us. FX_LOGS(INFO) << "Ready request sent"; client->Ready().ThenExactlyOnce( [&](fidl::WireUnownedResult<examples_canvas_clientrequesteddraw::Instance::Ready>& result) { // Check if the FIDL call succeeded or not. if (result.ok()) { FX_LOGS(INFO) << "Ready success"; } else { FX_LOGS(ERROR) << "Could not send Ready request: " << result.error(); } // Quit the loop, thereby handing control back to the outer loop of actions being // iterated over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); continue; } // Batch a line for drawing to the canvas using the two points provided. Line line = ParseLine(action); batched_lines.push_back(line); FX_LOGS(INFO) << "AddLines batching line: [Point { x: " << line[1].x << ", y: " << line[1].y << " }, Point { x: " << line[0].x << ", y: " << line[0].y << " }]"; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <fidl/examples.canvas.clientrequesteddraw/cpp/wire.h> #include <lib/async-loop/cpp/loop.h> #include <lib/async/cpp/task.h> #include <lib/component/outgoing/cpp/outgoing_directory.h> #include <lib/fidl/cpp/wire/channel.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <src/lib/fxl/macros.h> #include <src/lib/fxl/memory/weak_ptr.h> // A struct that stores the two things we care about for this example: the set of lines, and the // bounding box that contains them. struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. bool changed = true; // Tracks whether or not the client has declared itself ready to receive more updated. bool ready = true; examples_canvas_clientrequesteddraw::wire::BoundingBox bounding_box; }; // An implementation of the |Instance| protocol. class InstanceImpl final : public fidl::WireServer<examples_canvas_clientrequesteddraw::Instance> { public: // Bind this implementation to a channel. InstanceImpl(async_dispatcher_t* dispatcher, fidl::ServerEnd<examples_canvas_clientrequesteddraw::Instance> server_end) : binding_(dispatcher, std::move(server_end), this, std::mem_fn(&InstanceImpl::OnFidlClosed)), weak_factory_(this) { // Start the update timer on startup. Our server sends one update per second ScheduleOnDrawnEvent(dispatcher, zx::sec(1)); } void OnFidlClosed(fidl::UnbindInfo info) { if (info.reason() != ::fidl::Reason::kPeerClosedWhileReading) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; } void AddLines(AddLinesRequestView request, AddLinesCompleter::Sync& completer) override { FX_LOGS(INFO) << "AddLines request received"; for (const auto& points : request->lines) { FX_LOGS(INFO) << "AddLines printing line: [Point { x: " << points[1].x << ", y: " << points[1].y << " }, Point { x: " << points[0].x << ", y: " << points[0].y << " }]"; // Update the bounding box to account for the new line we've just "added" to the canvas. auto& bounds = state_.bounding_box; for (const auto& point : points) { if (point.x < bounds.top_left.x) { bounds.top_left.x = point.x; } if (point.y > bounds.top_left.y) { bounds.top_left.y = point.y; } if (point.x > bounds.bottom_right.x) { bounds.bottom_right.x = point.x; } if (point.y < bounds.bottom_right.y) { bounds.bottom_right.y = point.y; } } } // Mark the state as "dirty", so that an update is sent back to the client on the next |OnDrawn| // event. state_.changed = true; } void Ready(ReadyCompleter::Sync& completer) override { FX_LOGS(INFO) << "Ready request received"; // The client must only call `Ready() -> ();` after receiving an `-> OnDrawn();` event; if two // "consecutive" `Ready() -> ();` calls are received, this interaction has entered an invalid // state, and should be aborted immediately. if (state_.ready == true) { FX_LOGS(ERROR) << "Invalid back-to-back `Ready` requests received"; } state_.ready = true; completer.Reply(); } void handle_unknown_method( fidl::UnknownMethodMetadata<examples_canvas_clientrequesteddraw::Instance> metadata, fidl::UnknownMethodCompleter::Sync& completer) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << metadata.method_ordinal; } private: // Each scheduled update waits for the allotted amount of time, sends an update if something has // changed, and schedules the next update. void ScheduleOnDrawnEvent(async_dispatcher_t* dispatcher, zx::duration after) { async::PostDelayedTask( dispatcher, [&, dispatcher, after, weak = weak_factory_.GetWeakPtr()] { // Halt execution if the binding has been deallocated already. if (!weak) { return; } // Schedule the next update if the binding still exists. weak->ScheduleOnDrawnEvent(dispatcher, after); // No need to send an update if nothing has changed since the last one, or the client has // not yet informed us that it is ready for more updates. if (!weak->state_.changed || !weak->state_.ready) { return; } // This is where we would draw the actual lines. Since this is just an example, we'll // avoid doing the actual rendering, and simply send the bounding box to the client // instead. auto top_left = weak->state_.bounding_box.top_left; auto bottom_right = weak->state_.bounding_box.bottom_right; fidl::Status status = fidl::WireSendEvent(weak->binding_)->OnDrawn(top_left, bottom_right); if (!status.ok()) { return; } FX_LOGS(INFO) << "OnDrawn event sent: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; // Reset the change and ready trackers. state_.ready = false; weak->state_.changed = false; }, after); } fidl::ServerBinding<examples_canvas_clientrequesteddraw::Instance> binding_; CanvasState state_ = CanvasState{}; // Generates weak references to this object, which are appropriate to pass into asynchronous // callbacks that need to access this object. The references are automatically invalidated // if this object is destroyed. fxl::WeakPtrFactory<InstanceImpl> weak_factory_; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from the // client. The following initializes the loop, and obtains the dispatcher, which will be used when // binding the server implementation to a channel. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. This // directory is where the outgoing FIDL protocols are installed so that they can be provided to // other components. component::OutgoingDirectory outgoing = component::OutgoingDirectory(dispatcher); // The `ServeFromStartupInfo()` function sets up the outgoing directory with the startup handle. // The startup handle is a handle provided to every component by the system, so that they can // serve capabilities (e.g. FIDL protocols) to other components. zx::result result = outgoing.ServeFromStartupInfo(); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to serve outgoing directory: " << result.status_string(); return -1; } // Register a handler for components trying to connect to // |examples.canvas.clientrequesteddraw.Instance|. result = outgoing.AddUnmanagedProtocol<examples_canvas_clientrequesteddraw::Instance>( [dispatcher](fidl::ServerEnd<examples_canvas_clientrequesteddraw::Instance> server_end) { // Create an instance of our InstanceImpl that destroys itself when the connection closes. new InstanceImpl(dispatcher, std::move(server_end)); }); if (result.is_error()) { FX_LOGS(ERROR) << "Failed to add Instance protocol: " << result.status_string(); return -1; } // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
HLCPP
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <lib/async-loop/cpp/loop.h> #include <lib/sys/cpp/component_context.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <charconv> #include <examples/canvas/clientrequesteddraw/cpp/fidl.h> #include <examples/fidl/new/canvas/client_requested_draw/hlcpp/client/config.h> // A helper function that takes a coordinate in string form, like "123,-456", and parses it into a // a struct of the form |{ in64 x; int64 y; }|. ::examples::canvas::clientrequesteddraw::Point ParsePoint(std::string_view input) { int64_t x = 0; int64_t y = 0; size_t index = input.find(','); if (index != std::string::npos) { std::from_chars(input.data(), input.data() + index, x); std::from_chars(input.data() + index + 1, input.data() + input.length(), y); } return ::examples::canvas::clientrequesteddraw::Point{.x = x, .y = y}; } using Line = ::std::array<::examples::canvas::clientrequesteddraw::Point, 2>; // A helper function that takes a coordinate pair in string form, like "1,2:-3,-4", and parses it // into an array of 2 |Point| structs. Line ParseLine(const std::string& action) { auto input = std::string_view(action); size_t index = input.find(':'); if (index != std::string::npos) { return {ParsePoint(input.substr(0, index)), ParsePoint(input.substr(index + 1))}; } return {}; } int main(int argc, const char** argv) { FX_LOGS(INFO) << "Started"; // Retrieve component configuration. auto conf = config::Config::TakeFromStartupHandle(); // Start up an async loop. async::Loop loop(&kAsyncLoopConfigNeverAttachToThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Connect to the protocol inside the component's namespace, then create an asynchronous client // using the newly-established connection. examples::canvas::clientrequesteddraw::InstancePtr instance_proxy; auto context = sys::ComponentContext::Create(); context->svc()->Connect(instance_proxy.NewRequest(dispatcher)); FX_LOGS(INFO) << "Outgoing connection enabled"; instance_proxy.set_error_handler([&loop](zx_status_t status) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; loop.Quit(); }); // Provide a lambda to handle incoming |OnDrawn| events asynchronously. instance_proxy.events().OnDrawn = [&loop](::examples::canvas::clientrequesteddraw::Point top_left, ::examples::canvas::clientrequesteddraw::Point bottom_right) { FX_LOGS(INFO) << "OnDrawn event received: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; loop.Quit(); }; instance_proxy.events().handle_unknown_event = [](uint64_t ordinal) { FX_LOGS(WARNING) << "Received an unknown event with ordinal " << ordinal; }; std::vector<Line> batched_lines; for (const auto& action : conf.script()) { // If the next action in the script is to "PUSH", send a batch of lines to the server. if (action == "PUSH") { instance_proxy->AddLines(batched_lines); batched_lines.clear(); FX_LOGS(INFO) << "AddLines request sent"; continue; } // If the next action in the script is to "WAIT", block until an |OnDrawn| event is received // from the server. if (action == "WAIT") { loop.Run(); loop.ResetQuit(); // Now, inform the server that we are ready to receive more updates whenever they are ready // for us. FX_LOGS(INFO) << "Ready request sent"; instance_proxy->Ready([&](fpromise::result<void, fidl::FrameworkErr> result) { if (result.is_error()) { // Check that our flexible two-way call was known to the server and handle the case of an // unknown method appropriately. In the case of this example, there is nothing we can do // to recover here, except to log an error and exit the program. FX_LOGS(ERROR) << "Server does not implement AddLine"; } FX_LOGS(INFO) << "Ready success"; // Quit the loop, thereby handing control back to the outer loop of actions being iterated // over. loop.Quit(); }); // Run the loop until the callback is resolved, at which point we can continue from here. loop.Run(); loop.ResetQuit(); continue; } // Batch a line for drawing to the canvas using the two points provided. Line line = ParseLine(action); batched_lines.push_back(line); FX_LOGS(INFO) << "AddLines batching line: [Point { x: " << line[1].x << ", y: " << line[1].y << " }, Point { x: " << line[0].x << ", y: " << line[0].y << " }]"; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. sleep(2); return 0; }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include <lib/async-loop/cpp/loop.h> #include <lib/async-loop/default.h> #include <lib/async/cpp/task.h> #include <lib/fidl/cpp/binding.h> #include <lib/sys/cpp/component_context.h> #include <lib/syslog/cpp/macros.h> #include <unistd.h> #include <examples/canvas/clientrequesteddraw/cpp/fidl.h> #include <src/lib/fxl/macros.h> #include <src/lib/fxl/memory/weak_ptr.h> // A struct that stores the two things we care about for this example: the set of lines, and the // bounding box that contains them. struct CanvasState { // Tracks whether there has been a change since the last send, to prevent redundant updates. bool changed = true; // Tracks whether or not the client has declared itself ready to receive more updated. bool ready = true; examples::canvas::clientrequesteddraw::BoundingBox bounding_box; }; using Line = ::std::array<::examples::canvas::clientrequesteddraw::Point, 2>; // An implementation of the |Instance| protocol. class InstanceImpl final : public examples::canvas::clientrequesteddraw::Instance { public: // Bind this implementation to an |InterfaceRequest|. InstanceImpl(async_dispatcher_t* dispatcher, fidl::InterfaceRequest<examples::canvas::clientrequesteddraw::Instance> request) : binding_(fidl::Binding<examples::canvas::clientrequesteddraw::Instance>(this)), weak_factory_(this) { binding_.Bind(std::move(request), dispatcher); // Gracefully handle abrupt shutdowns. binding_.set_error_handler([this](zx_status_t status) mutable { if (status != ZX_ERR_PEER_CLOSED) { FX_LOGS(ERROR) << "Shutdown unexpectedly"; } delete this; }); // Start the update timer on startup. Our server sends one update per second. ScheduleOnDrawnEvent(dispatcher, zx::sec(1)); } void AddLines(std::vector<Line> lines) override { FX_LOGS(INFO) << "AddLines request received"; for (const auto& points : lines) { FX_LOGS(INFO) << "AddLines printing line: [Point { x: " << points[1].x << ", y: " << points[1].y << " }, Point { x: " << points[0].x << ", y: " << points[0].y << " }]"; // Update the bounding box to account for the new line we've just "added" to the canvas. auto& bounds = state_.bounding_box; for (const auto& point : points) { if (point.x < bounds.top_left.x) { bounds.top_left.x = point.x; } if (point.y > bounds.top_left.y) { bounds.top_left.y = point.y; } if (point.x > bounds.bottom_right.x) { bounds.bottom_right.x = point.x; } if (point.y < bounds.bottom_right.y) { bounds.bottom_right.y = point.y; } } } // Mark the state as "dirty", so that an update is sent back to the client on the next // |OnDrawn| event. state_.changed = true; } void Ready(ReadyCallback callback) override { FX_LOGS(INFO) << "Ready request received"; // The client must only call `Ready() -> ();` after receiving an `-> OnDrawn();` event; if // two "consecutive" `Ready() -> ();` calls are received, this interaction has entered an // invalid state, and should be aborted immediately. if (state_.ready == true) { FX_LOGS(ERROR) << "Invalid back-to-back `Ready` requests received"; } state_.ready = true; callback(fpromise::ok()); } void handle_unknown_method(uint64_t ordinal, bool method_has_response) override { FX_LOGS(WARNING) << "Received an unknown method with ordinal " << ordinal; } private: // Each scheduled update waits for the allotted amount of time, sends an update if something // has changed, and schedules the next update. void ScheduleOnDrawnEvent(async_dispatcher_t* dispatcher, zx::duration after) { async::PostDelayedTask( dispatcher, [&, dispatcher, after, weak = weak_factory_.GetWeakPtr()] { // Halt execution if the binding has been deallocated already. if (!weak) { return; } // Schedule the next update if the binding still exists. weak->ScheduleOnDrawnEvent(dispatcher, after); // No need to send an update if nothing has changed since the last one, or the client // has not yet informed us that it is ready for more updates. if (!weak->state_.changed || !weak->state_.ready) { return; } // This is where we would draw the actual lines. Since this is just an example, we'll // avoid doing the actual rendering, and simply send the bounding box to the client // instead. auto top_left = state_.bounding_box.top_left; auto bottom_right = state_.bounding_box.bottom_right; binding_.events().OnDrawn(top_left, bottom_right); FX_LOGS(INFO) << "OnDrawn event sent: top_left: Point { x: " << top_left.x << ", y: " << top_left.y << " }, bottom_right: Point { x: " << bottom_right.x << ", y: " << bottom_right.y << " }"; // Reset the change and ready trackers. state_.ready = false; state_.changed = false; }, after); } fidl::Binding<examples::canvas::clientrequesteddraw::Instance> binding_; CanvasState state_ = CanvasState{}; // Generates weak references to this object, which are appropriate to pass into asynchronous // callbacks that need to access this object. The references are automatically invalidated // if this object is destroyed. fxl::WeakPtrFactory<InstanceImpl> weak_factory_; }; int main(int argc, char** argv) { FX_LOGS(INFO) << "Started"; // The event loop is used to asynchronously listen for incoming connections and requests from // the client. The following initializes the loop, and obtains the dispatcher, which will be // used when binding the server implementation to a channel. // // Note that unlike the new C++ bindings, HLCPP bindings rely on the async loop being attached // to the current thread via the |kAsyncLoopConfigAttachToCurrentThread| configuration. async::Loop loop(&kAsyncLoopConfigAttachToCurrentThread); async_dispatcher_t* dispatcher = loop.dispatcher(); // Create an |OutgoingDirectory| instance. // // The |component::OutgoingDirectory| class serves the outgoing directory for our component. // This directory is where the outgoing FIDL protocols are installed so that they can be // provided to other components. auto context = sys::ComponentContext::CreateAndServeOutgoingDirectory(); // Register a handler for components trying to connect to // |examples.canvas.clientrequesteddraw.Instance|. context->outgoing()->AddPublicService( fidl::InterfaceRequestHandler<examples::canvas::clientrequesteddraw::Instance>( [dispatcher]( fidl::InterfaceRequest<examples::canvas::clientrequesteddraw::Instance> request) { // Create an instance of our |InstanceImpl| that destroys itself when the connection // closes. new InstanceImpl(dispatcher, std::move(request)); })); // Everything is wired up. Sit back and run the loop until an incoming connection wakes us up. FX_LOGS(INFO) << "Listening for incoming connections"; loop.Run(); return 0; }
Union payload
FIDL recipe: Union payload
A union payload is a FIDL method payload that uses the union layout. The
top-level type used as the method payload must use one of struct, table, or
union as its layout. Notably, some generated bindings "flatten" the arguments
passed to struct method payloads, such that each member is itself treated as a
function argument in the calling signature. Payloads that use table or union
never do this, and always pass a single argument, called payload, instead.
Reasoning
The key-value store baseline example's implementation was a good starting point, but one major drawback is that data is stored as raw bytes. FIDL is a richly typed language. Forcing data that is for instance a UTF-8 string to be stored as an untyped byte array erases this valuable type information for readers of the *.fidl file, as well as for programmers using bindings generated from it.
Implementation
The main goal of this change is to replace the baseline case's vector<byte>
typed value member with a union that stores many possible types. In fact, as
of this change a good survey of FIDL's
value types is on
offer:
- All of FIDL's builtin scalar types are used as variants in the
Valueunion:bool,uint8,uint16,uint32,uint64,int8,int16,int32,int64,float32, andfloat64(also known as FIDL's primitive types), as well asstring. - This
unionalso features uses of FIDL's builtinarray<T, N>andvector<T>type templates. - All of FIDL's type layouts, namely
bits,enum,table,union, andstruct, are utilized in this example at least once.
The request and response payloads used for WriteItem have also been changed
from structs to a named table and an inlined flexible union, respectively.
In fact, any of these three layouts may be used a request/response payload. The
latter two, known as table payloads and *union payloads, respectively, are
preferred in all but the most message size sensitive cases. This is because they
are much easier to extend in the future in a binary compatible way.
FIDL
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. library examples.keyvaluestore.usegenericvalues; /// An item in the store. The key must match the regex `^[A-z][A-z0-9_\.\/]{2,62}[A-z0-9]$`. That /// is, it must start with a letter, end with a letter or number, contain only letters, numbers, /// periods, and slashes, and be between 4 and 64 characters long. type Item = struct { key string:128; value Value; }; // Because the `Value` must be used both in the request and the response, we give it its own named // type. The type is a `union` of all possible data types that we take as values, and is marked // `flexible` to allow for the easy addition of new data types in the future. type Value = flexible union { // Keep the original `bytes` as one of the options in the new union. 1: bytes vector<byte>:64000; // A `string` is very similar to `vector<byte>` on the wire, with the extra constraint that // it enforces that it enforces that the byte vector in question is valid UTF-8. 2: string string:64000; // All of FIDL's primitive types. 3: bool bool; 4: uint8 uint8; 5: int8 int8; 6: uint16 uint16; 7: int16 int16; 8: uint32 uint32; 9: int32 int32; 10: float32 float32; 11: uint64 uint64; 12: int64 int64; 13: float64 float64; // FIDL does not natively support 128-bit integer types, so we have to define our own // representations. 14: uint128 array<uint64, 2>; }; // Because we now supoprt a richer range of types as values in our store, it is helpful to use a // `flexible`, and therefore evolvable, `bits` type to store write options. type WriteOptions = flexible bits : uint8 { // This flag allows us to overwrite existing data when there is a collision, rather than failing // with an `WriteError.ALREADY_EXISTS`. OVERWRITE = 0b1; // This flag allows us to concatenate to existing data when there is a collision, rather than // failing with an `WriteError.ALREADY_EXISTS`. "Concatenation" means addition for the numeric // variants and appending to the `bytes`/`string` variants. If no existing data can be found, we // "concatenate" to default values of zero and an empty vector, respectively. Attempting to // concatenate to an existing variant of a different type will return a // `WriteError.INVALID_VALUE` error. CONCAT = 0b10; }; /// An enumeration of things that may go wrong when trying to write a value to our store. type WriteError = flexible enum { UNKNOWN = 0; INVALID_KEY = 1; INVALID_VALUE = 2; ALREADY_EXISTS = 3; }; /// A very basic key-value store. @discoverable open protocol Store { /// Writes an item to the store. /// /// Since the value stored in the key-value store can now be different from the input (if the /// `WriteOptions.CONCAT` flag is set), we need to return the resulting `Value` to the /// requester. /// /// We use an (anonymous) `table` and a (named) `flexible union` as the request and response /// payload, respectively, to allow for easier future evolution. Both of these types are /// `flexible`, meaning that adding or removing members is binary-compatible. This makes them /// much easier to evolve that the `struct` types that were previously used, which cannot be /// changed after release without breaking ABI. flexible WriteItem(table { 1: attempt Item; 2: options WriteOptions; }) -> (Value) error WriteError; };
CML
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/client_bin", }, use: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store" }, ], config: { // A vector of values for every easily representible type in our key-value store. For // brevity's sake, the 8, 16, and 32 bit integer types and booleans are omitted. // // TODO(https://fxbug.dev/42178362): It would absolve individual language implementations of a great // deal of string parsing if we were able to use all FIDL constructs directly here. In // particular, floats and nested types are very difficult to represent, and have been // excluded from this example for the time being. set_concat_option: { type: "bool" }, set_overwrite_option: { type: "bool" }, write_bytes: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, write_strings: { type: "vector", max_count: 16, element: { type: "string", max_size: 64, }, }, write_uint64s: { type: "vector", max_count: 16, element: { type: "uint64" }, }, write_int64s: { type: "vector", max_count: 16, element: { type: "int64" }, }, // Note: due to the limitation of structured config not allowing vectors nested in vectors, // we only set the lower half of the uint128 for simplicity's sake. write_uint128s: { type: "vector", max_count: 16, element: { type: "uint64" }, }, }, }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { include: [ "syslog/client.shard.cml" ], program: { runner: "elf", binary: "bin/server_bin", }, capabilities: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store" }, ], expose: [ { protocol: "examples.keyvaluestore.usegenericvalues.Store", from: "self", }, ], }
Realm
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. { children: [ { name: "client", url: "#meta/client.cm", }, { name: "server", url: "#meta/server.cm", }, ], offer: [ // Route the protocol under test from the server to the client. { protocol: "examples.keyvaluestore.usegenericvalues.Store", from: "#server", to: "#client", }, { dictionary: "diagnostics", from: "parent", to: "all", }, // Route diagnostics support to all children. { protocol: [ "fuchsia.inspect.InspectSink", "fuchsia.logger.LogSink", ], from: "parent", to: [ "#client", "#server", ], }, ], }
Client and server implementations can then be written in any supported language:
Rust
Client
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use { anyhow::{Context as _, Error}, config::Config, fidl_examples_keyvaluestore_usegenericvalues::{ Item, StoreMarker, StoreProxy, StoreWriteItemRequest, Value, WriteOptions, }, fuchsia_component::client::connect_to_protocol, std::{thread, time}, }; // A helper function to sequentially write a single item to the key-value store and print a log when // successful. async fn write_next_item( store: &StoreProxy, key: &str, value: Value, options: WriteOptions, ) -> Result<(), Error> { // Create an empty request payload using `::default()`. let mut req = StoreWriteItemRequest::default(); req.options = Some(options); // Fill in the `Item` we will be attempting to write. println!("WriteItem request sent: key: {}, value: {:?}", &key, &value); req.attempt = Some(Item { key: key.to_string(), value: value }); // Send and async `WriteItem` request to the server. match store.write_item(&req).await.context("Error sending request")? { Ok(value) => println!("WriteItem response received: {:?}", &value), Err(err) => println!("WriteItem Error: {}", err.into_primitive()), } Ok(()) } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Load the structured config values passed to this component at startup. let config = Config::take_from_startup_handle(); // Use the Component Framework runtime to connect to the newly spun up server component. We wrap // our retained client end in a proxy object that lets us asynchronously send `Store` requests // across the channel. let store = connect_to_protocol::<StoreMarker>()?; println!("Outgoing connection enabled"); // All of our requests will have the same bitflags set. Pull these settings from the config. let mut options = WriteOptions::empty(); options.set(WriteOptions::OVERWRITE, config.set_overwrite_option); options.set(WriteOptions::CONCAT, config.set_concat_option); // The structured config provides one input for most data types that can be stored in the data // store. Iterate through those inputs in the order we see them in the FIDL file. // // Note that FIDL unions are rendered as enums in Rust; for example, the `Value` union has now // become a `Value` Rust enum, with each member taking exactly one argument. for value in config.write_bytes.into_iter() { write_next_item(&store, "bytes", Value::Bytes(value.into()), options).await?; } for value in config.write_strings.into_iter() { write_next_item(&store, "string", Value::String(value), options).await?; } for value in config.write_uint64s.into_iter() { write_next_item(&store, "uint64", Value::Uint64(value), options).await?; } for value in config.write_int64s.into_iter() { write_next_item(&store, "int64", Value::Int64(value), options).await?; } for value in config.write_uint128s.into_iter() { write_next_item(&store, "uint128", Value::Uint128([0, value]), options).await?; } // TODO(https://fxbug.dev/42156498): We need to sleep here to make sure all logs get drained. Once the // referenced bug has been resolved, we can remove the sleep. thread::sleep(time::Duration::from_secs(2)); Ok(()) }
Server
// Copyright 2022 The Fuchsia Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. use anyhow::{Context as _, Error}; use fuchsia_component::server::ServiceFs; use futures::prelude::*; use lazy_static::lazy_static; use regex::Regex; use std::cell::RefCell; use std::collections::hash_map::Entry; use std::collections::HashMap; use fidl_examples_keyvaluestore_usegenericvalues::{ Item, StoreRequest, StoreRequestStream, Value, WriteError, WriteOptions, }; use std::collections::hash_map::OccupiedEntry; use std::ops::Add; lazy_static! { static ref KEY_VALIDATION_REGEX: Regex = Regex::new(r"^[A-Za-z]\w+[A-Za-z0-9]$").expect("Key validation regex failed to compile"); } /// Sums any numeric type. fn sum<T: Add + Add<Output = T> + Copy>(operands: [T; 2]) -> T { operands[0] + operands[1] } /// Clones and inserts an entry, so that the original (now concatenated) copy may be returned in the /// response. fn write(inserting: Value, mut entry: OccupiedEntry<'_, String, Value>) -> Value { entry.insert(inserting.clone()); println!("Wrote key: {}, value: {:?}", entry.key(), &inserting); inserting } /// Handler for the `WriteItem` method. fn write_item( store: &mut HashMap<String, Value>, attempt: Item, options: &WriteOptions, ) -> Result<Value, WriteError> { // Validate the key. if !KEY_VALIDATION_REGEX.is_match(attempt.key.as_str()) { println!("Write error: INVALID_KEY for key: {}", attempt.key); return Err(WriteError::InvalidKey); } match store.entry(attempt.key) { Entry::Occupied(entry) => { // The `CONCAT` flag supersedes the `OVERWRITE` flag, so check it first. if options.contains(WriteOptions::CONCAT) { match entry.get() { Value::Bytes(old) => { if let Value::Bytes(new) = attempt.value { let mut combined = old.clone(); combined.extend(new); return Ok(write(Value::Bytes(combined), entry)); } } Value::String(old) => { if let Value::String(new) = attempt.value { return Ok(write(Value::String(format!("{}{}", old, &new)), entry)); } } Value::Uint64(old) => { if let Value::Uint64(new) = attempt.value { return Ok(write(Value::Uint64(sum([*old, new])), entry)); } } Value::Int64(old) => { if let Value::Int64(new) = attempt.value { return Ok(write(Value::Int64(sum([*old, new])), entry)); } } // Note: only works on the uint64 range in practice. Value::Uint128(old) => { if let Value::Uint128(new) = attempt.value { return Ok(write(Value::Uint128([0, sum([old[1], new[1]])]), entry)); } } _ => { panic!("actively unsupported type!") } } // Only reachable if the type of the would be concatenated value did not match the // value already occupying this entry. println!("Write error: INVALID_VALUE for key: {}", entry.key()); return Err(WriteError::InvalidValue); } // If we're not doing CONCAT, check for OVERWRITE next. if options.contains(WriteOptions::OVERWRITE) { return Ok(write(attempt.value, entry)); } println!("Write error: ALREADY_EXISTS for key: {}", entry.key()); Err(WriteError::AlreadyExists) } Entry::Vacant(entry) => { println!("Wrote key: {}, value: {:?}", entry.key(), &attempt.value); entry.insert(attempt.value.clone()); Ok(attempt.value) } } } /// Creates a new instance of the server. Each server has its own bespoke, per-connection instance /// of the key-value store. async fn run_server(stream: StoreRequestStream) -> Result<(), Error> { // Create a new in-memory key-value store. The store will live for the lifetime of the // connection between the server and this particular client. let store = RefCell::new(HashMap::<String, Value>::new()); // Serve all requests on the protocol sequentially - a new request is not handled until its // predecessor has been processed. stream .map(|result| result.context("failed request")) .try_for_each(|request| async { // Match based on the method being invoked. match request { // Because we are using a table payload, there is an extra level of indirection. The // top-level container for the table itself is always called "payload". StoreRequest::WriteItem { payload, responder } => { println!("WriteItem request received"); // Error out if either of the request table's members are not set. let attempt = payload.attempt.context("required field 'attempt' is unset")?; let options = payload.options.context("required field 'options' is unset")?; // The `responder` parameter is a special struct that manages the outgoing reply // to this method call. Calling `send` on the responder exactly once will send // the reply. responder .send( write_item(&mut store.borrow_mut(), attempt, &options) .as_ref() .map_err(|e| *e), ) .context("error sending reply")?; println!("WriteItem response sent"); } // StoreRequest::_UnknownMethod { ordinal, .. } => { println!("Received an unknown method with ordinal {ordinal}"); } } Ok(()) }) .await } // A helper enum that allows us to treat a `Store` service instance as a value. enum IncomingService { Store(StoreRequestStream), } #[fuchsia::main] async fn main() -> Result<(), Error> { println!("Started"); // Add a discoverable instance of our `Store` protocol - this will allow the client to see the // server and connect to it. let mut fs = ServiceFs::new_local(); fs.dir("svc").add_fidl_service(IncomingService::Store); fs.take_and_serve_directory_handle()?; println!("Listening for incoming connections"); // The maximum number of concurrent clients that may be served by this process. const MAX_CONCURRENT: usize = 10; // Serve each connection simultaneously, up to the `MAX_CONCURRENT` limit. fs.for_each_concurrent(MAX_CONCURRENT, |IncomingService::Store(stream)| { run_server(stream).unwrap_or_else(|e| println!("{:?}", e)) }) .await; Ok(()) }
C++ (Natural)
Client
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Natural) implementation.C++ (Wire)
Client
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.Server
// TODO(https://fxbug.dev/42060656): C++ (Wire) implementation.HLCPP
Client
// TODO(https://fxbug.dev/42060656): HLCPP implementation.Server
// TODO(https://fxbug.dev/42060656): HLCPP implementation.