Execute a series of job flows. More...

Detailed Description

Very (very) simple flow manager. Jobs execute in sequential waves, all previous waves must finish successfully.

The input table is formed as per below. Each observation represents one job. Each variable is converted into a macro variable with the same name.

Input table (minimum variables needed)

_PROGRAM - Provides the path to the job itself
FLOW_ID - Numeric value, provides sequential ordering capability. Is optional, will default to 0 if not provided.
_CONTEXTNAME - Dictates which context should be used to run the job. If blank, or not provided, will default to SAS Job Execution compute context.

Any additional variables provided in this table are converted into macro variables and passed into the relevant job.

_PROGRAM	FLOW_ID (optional)	_CONTEXTNAME (optional)
/Public/jobs/somejob1	0	SAS Job Execution compute context
/Public/jobs/somejob2	0	SAS Job Execution compute context

Output table (minimum variables produced)

_PROGRAM - the SAS Drive path of the job
URI - the URI of the executed job
STATE - the completed state of the job
TIMESTAMP - the datetime that the job completed
JOBPARAMS - the parameters that were passed to the job
FLOW_ID - the id of the flow in which the job was executed

To avoid hammering the box with many hits in rapid succession, a one second pause is made between every request.

Example

First, compile the macros:

filename mc url
"https://raw.githubusercontent.com/sasjs/core/main/all.sas";
%inc mc;

Next, create some jobs (in this case, as web services):

filename ft15f001 temp;
parmcards4;
  %put this is job: &_program;
  %put this was run in flow &flow_id;
  data ;
    rand=ranuni(0)*&macrovar1;
    do x=1 to rand;
      y=rand*&macrovar2;
      if y=100 then abort;
      output;
    end;
  run;
;;;;
%mv_createwebservice(path=/Public/temp,name=demo1)
%mv_createwebservice(path=/Public/temp,name=demo2)

Prepare an input table with 60 executions:

data work.inputjobs;
  _contextName='SAS Job Execution compute context';
  do flow_id=1 to 3;
    do i=1 to 20;
      _program='/Public/temp/demo1';
      macrovar1=10*i;
      macrovar2=4*i;
      output;
      i+1;
      _program='/Public/temp/demo2';
      macrovar1=40*i;
      macrovar2=44*i;
      output;
    end;
  end;
run;

Trigger the flow

%mv_jobflow(inds=work.inputjobs
  ,maxconcurrency=4
  ,outds=work.results
  ,outref=myjoblog
)

data _null_;
  infile myjoblog;
  input; put _infile_;
run;

Parameters

[in]	access_token_var=	The global macro variable to contain the access token
[in]	grant_type=	valid values: password authorization_code detect - will check if access_token exists, if not will use sas_services if a SASStudioV session else authorization_code. Default option. sas_services - will use oauth_bearer=sas_services
[in]	inds=	The input dataset containing a list of jobs and parameters
[in]	maxconcurrency=	The max number of parallel jobs to run. Default=8.
[in]	raise_err=0	Set to 1 to raise SYSCC when a job does not complete succcessfully
[in]	mdebug=	set to 1 to enable DEBUG messages
[out]	outds=	The output dataset containing the results
[out]	outref=	The output fileref to which to append the log file(s).

Version: VIYA V.03.05

Author: Allan Bowe, source: https://github.com/sasjs/core

SAS Macros

Definition in file mv_jobflow.sas.