Create Job Definitions

Lakehousecat offers a robust framework for defining various jobs, such as automation processes, backups, data refreshes, and semantic processes to construct semantic layers. While many predefined job definitions are available within Lakehousecat's framework, users have the flexibility to create their own custom job definitions. This documentation provides a detailed guide on crafting job definitions independently.

Overview

Job Definitions in Lakehousecat are configurations that specify various operational tasks to be executed automatically or on a schedule. By creating custom job definitions, users can tailor operations to meet specific needs, whether they're related to data synchronization, administrative tasks, or other processes.

Prerequisites

Before creating job definitions, ensure you have:

• Access as an administrator or with a builder role in Lakehousecat.
• Familiarity with Lakehousecat’s workspace and navigation.

Step-by-Step Guide

1. Access Workspace

   • Log into Lakehousecat and navigate to your designated workspace.

2. Navigate to Operations Section

   • Find the 'Operations' section and click to view various operational options within Lakehousecat.

3. Access Job Definitions

   • Within the Operations section, click on 'Job Definitions' to begin.

4. Create a New Job Definition

   • Click the Plus icon to initiate the creation of a new Job Definition.

5. Enter Job Definition Details

   • Unique Name: Choose a unique name following naming conventions for easy identification.
   • Description (Optional): Add a description to clarify the job’s purpose.
   • Job Type: Select an appropriate Job Type. Options include:
     • Admin-Task: For administrative operations.
     • Incremental or Full-Load Data Source: For loading processes.
     • Data-Sync and Admin-Task: Common types for synchronization and upkeep tasks.

6. Configure Scheduling

   • Define a schedule using cron syntax to automate the execution of tasks.

7. Set Additional Properties

   • Enabled: Toggle to activate the job.
   • Visible: Determine its visibility in the system.
   • Locked: Indicates if the job definition is locked from edits.

Defining Tasks Within Job Definitions

Each Job Definition should consist of at least one task. Tasks can be configured for:

• Worker-Type Selection: Choose from Admin, Builder, Semantic, or User types to grant specific permissions.

• Task-ID: Automatically generated for each task.

• Resource Specification: Define resources required from the Kubernetes cluster.

• Advanced Settings: Configure environment variables and secrets for task operation.

Payload Creation

• Execution Script: Develop a Shell script named run.sh as the entry point for execution. This script should contain all necessary instructions for task execution.
• Test Scripts: Consider creating test scripts to demonstrate the operation of the job.

Task Networking and Management

• Define multiple tasks within a job definition, allowing for a comprehensive representation of operations and resource connections.

Best Practices

• Consistency in Naming: Follow naming conventions to maintain clarity across job definitions.
• Detailed Descriptions: Provide detailed descriptions, especially in complex jobs with multiple tasks.

Troubleshooting

Creating Job Definitions is complex and intended primarily for administrators and builders. Monitor and test each job definition following creation to ensure functionality and address any issues promptly.

This MDX documentation will equip you to successfully create and manage Job Definitions within Lakehousecat, enhancing operational efficiency through customized task automation.

Please ensure this documentation remains comprehensive and clear, offering substantial guidance to new and experienced users alike.