Sherpa

Introduction

You Sudo You™

Built for the sudoer in all of us, Sherpa lets you blend URIs, APIs, CLI commands, and script blocks into reusable Runbooks, letting you dev/test/ops your way to success.

With Great Power…

If you’ve worked with the Cloud for any period of time you’ve likely come to the realization that with great power comes great complexity.

By “complexity” we mean frustration, lost productivity, and pain supporting simple business requests the Cloud can make far from simple.

We mean countless hours lost clicking through ever-changing web dashboards that require so many clicks, tabs, and transitions you’d swear they were designed to maximize page views not productivity.

We mean countless more searching meager documentation and myriad StackOverflow threads to track down the specific flavor of flag soup you need.

We mean yet more hours coding boilerplate logic to handle next-batch tokens, cursors, file streams, and buffers, just so you can begin to process data from services that only communicate in asynchronous chunks.

Sherpa fixes all that with a powerful DevOps-oriented interface.

The REPL that ROCKS™

Web dashboards are fine for monitoring and one-off tasks but not much else.

Dashboards are constantly in flux as providers make changes and upgrades. For the most part you can’t access their data easily, automate through them, audit activities performed in them, or… well, you get the picture.

To do repeated tasks of any value you have two choices: CLI or SDK.

Whether you choose CLI or SDK you’re likely to find yourself using a REPL (Read-Eval-Print Loop) of sorts as you work out the details of the CLI command or SDK API you’re interacting with.

CLI interactions rely on terminals; essentially simple text REPLs for Shells such as sh, bash, or zsh. Text in, text out. That’s the full feature set.

SDK interactions, at least with interpreted languages like JavaScript, Python, et. al., imply a REPL interface equivalent to that of a terminal.

Sherpa transforms both experiences with a notebook-inspired interface that lets you interact with JavaScript, Shell, and other language interpreters in a single, unified, feature-rich environment.

Sherpa Inputs

Unlike a terminal:

Sherpa’s input cells default to compact multi-line editors which size to fit your input while supporting syntax checking and common key bindings across multiple input languages.
Each entry can target a different shell. Sherpa supports it’s own enhanced JavaScript-based shell (TSH), NodeJS, Chromium JS, and a subset of other shells accessible on your platform (sh, bash, etc).
Input cells can select from a variety of input editors. While multi-line editing is the default you can create your own input forms or other controls allowing you to fine-tune data input to the task at hand.

Sherpa Outputs

Unlike a terminal:

Sherpa’s output cells are XML/XHTML/SVG rendering surfaces. Sherpa’s design includes pre-built renderers for common data types including JSON and XML data and common data structures such as lists, tables, and trees.
Output cell data retains its object form. Text output in an output cell can be run through one or more “recognizers” to transform it from raw text into object form for easier manipulation and/or rendering.
Output cell data can be exported directly to a file or sent to target URLs. There’s no need for copy/paste, string escaping, etc. to access output.

Sherpa lets you enter data efficiently and visualize results effectively.

From REPL to Runbook

Working with URIs, JavaScript, or bash in a more iterative way is a great first step, however the goal is to make scripting and automating interactions easier, to reduce complexity.

Sherpa adds three powerful features to do just that:

CommandFlow™: Cross-command choreography and data pipelining.
Bullet Time™: Work “between the batches” with big data sets/files.
ScriptApps™: Convert CLI “flag soup” into easy-to-use input forms.

CommandFlow™

Working with Cloud CLIs and SDKs teaches you fairly quickly that some things are easier in the CLI and some are easier via the SDK.

Most tools make it difficult or impossible to blend CLI and SDK logic, preventing you from using the best tool for each job. Not Sherpa.

Leveraging JavaScript, Promises, and Sherpa’s underlying TIBET infrastructure, CommandFlow transforms what would otherwise be isolated code blocks into powerful, workflow-driven scripts we call Runbooks.

Imagine you entered the following AWS CLI command:

$ aws logs get-log-events — log-group-name $LOG_GROUP — log-stream-name `aws logs describe-log-streams — log-group-name $LOG_GROUP — max-items 1 — order-by LastEventTime — descending — query logStreams[].logStreamName — output text | head -n 1` — query events[].message — output text

Off it goes, fetching data. Once the call completes you want to pluck out the @message field containing the message text, potentially JSON.parse it to turn it into fields you can filter on, then summarize that result.

You could do all that via more Shell, but with Sherpa you can also do it via JavaScript simply by referencing the prior entry’s output:

Whenever you trigger an entry CommandFlow automatically wraps that entry in a Promise while marshaling inputs, mapping parameters, and performing other choreography so your Runbook entries work together.

CommandFlow transforms Sherpa from a simple scripting playground into a powerful workflow execution engine.

BulletTime™

Cloud (and database) APIs almost always return data in batches.

Query the Cloud and you get a batch of records and an optional next-batch token.

Query a database and you often need start/limit info and/or cursors.

File access is much the same.

Exceed NodeJS’s file size limit and you’ll be forced to rework code to use streaming APIs which feed data in chunks.

When working with data, particularly big data, problems that are conceptually simple suddenly get a lot more complicated due to async, batching, streaming, etc.

Time Management (Sherpa-style ;))

Sherpa’s built-in data fetch / data store Tasks are designed to be batch, cursor, and stream aware, allowing you to work with data without losing focus or flow.

When you invoke a data fetch built-in, Sherpa fetches the first batch/chunk of data, normalizes it, and makes it available to your next Task.

Regardless of how many iterations it takes to get your logic right, Sherpa spares you from repeated network delays, authentication refreshes, API invocations, and other productivity sinks.

Once you’ve got your logic working, Sherpa lets you “restart time”, fetching and processing your entire data set without ever having to think about batch tokens, streams, buffers, or other low-level details.

We call Sherpa’s ability to suspend time in this way BulletTime, an apt metaphor for letting you work in “the space between the batches”.

You stay focused on your core logic. Sherpa does the rest.

ScriptApps™

So, you’ve blended the best parts of your Cloud provider’s CLI and SDK interfaces using CommandFlow and worked with a single batch of log data to sort out your pluck, filter, summarize logic thanks to BulletTime.

Thing is, there’s still this complex command line lurking in your Runbook and changing the value for that LOG_GROUP variable can be error-prone:

$ aws logs get-log-events — log-group-name $LOG_GROUP — log-stream-name `aws logs describe-log-streams — log-group-name $LOG_GROUP — max-items 1 — order-by LastEventTime — descending — query logStreams[].logStreamName — output text | head -n 1` — query events[].message — output text

No problem.

Sherpa’s flexible input editors let you transform Runbooks from command-driven workflows into powerful micro-applications we call ScriptApps.

Sherpa takes flag soup and lets you turn it into forms:

From Runnable To Reusable

We’ve focused on the CLI and SDK execution portions of Sherpa Runbooks up to this point. If you’ve looked closely at the screenshots however you probably noticed they often include standalone text or todo items as well.

Text Entries

Text entries let you enter markdown or XHTML to serve as inline documentation for a runbook. You can have as many Text entries as you like and they can be mixed throughout your Runbook.

Text entries have no impact on the execution sequence of your Runbook.

Todo Entries

Todo entries let you create literal todo items at different points in your Runbook. In the example screenshots a common one is a todo encouraging you to "Verify your AWS Credentials" before proceeding.

Todo entries may impact the execution of your Runbook.

A Todo entry can be blocking or non-blocking meaning you have to actively check it off before the rest of the Runbook’s logic will proceed. By default Todo entries are blocking.

Task Entries

Sherpa includes a wide variety of Task entries. We’ve focused on the shell invocation type in this document but CommandFlow also provides a number of pre-built Task types supporting branching, looping, split/join logic, and other operations, allowing you to create complex workflows.

Purpose-Built or Pre-Built

If you’re looking to create your own Runbooks you can do that quickly and easily using Sherpa’s notebook-inspired REPL. Sherpa also allows you to share your purpose-built Runbooks securely among members of your team, creating the foundation for repeatable, predictable processes.

If you’re looking for out-of-the-box solutions, Sherpa provides you with access to a catalog of pre-built Runbooks created and tested by our team.

As shown above, examples include AWS Lambda deployment, comparing local and deployed Lambda versions, configuring the Lambda environment, querying CloudWatch logs, and more.

Organized around the DevOps lifecycle, Sherpa Runbooks can be grouped into Playbooks supporting a particular platform, project, or lifecycle phase. A Runbook can belong to any number of Playbooks.

Defining a Sherpa Project is another way of organizing Runbooks. Each Project gets its own Playbook allowing you to fine-tune the collection of Runbooks which apply to that Project:

Whether you’re using purpose-built or pre-built Runbooks, Sherpa helps you keep them organized and accessible.

Pup